Nov 03 16:11:33.576589 ip-10-0-139-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 16:11:33.576600 ip-10-0-139-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 16:11:33.576610 ip-10-0-139-190 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 16:11:33.576918 ip-10-0-139-190 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 16:11:43.693977 ip-10-0-139-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 16:11:43.693991 ip-10-0-139-190 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6b66123d86824e8283e81b8b25b10719 -- Nov 03 16:13:35.076990 ip-10-0-139-190 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:13:35.489484 ip-10-0-139-190 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:35.489484 ip-10-0-139-190 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:13:35.489484 ip-10-0-139-190 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:35.489484 ip-10-0-139-190 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:35.495034 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.494938 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.495034 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495030 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.495034 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495036 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.495034 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495040 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.495034 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495044 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495048 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495052 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495056 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495060 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495065 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495068 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495072 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495075 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495079 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495091 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495095 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495099 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495102 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495106 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495110 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495114 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495117 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495121 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495125 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.495339 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495129 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495133 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495137 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495141 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495145 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495149 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495153 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495156 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495160 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495164 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495168 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495173 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495177 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495182 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495187 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495191 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495196 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495201 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495206 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495210 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.496167 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495215 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495220 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495224 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495229 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495234 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495238 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495242 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495248 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495252 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495256 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495260 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495264 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495268 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495273 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495277 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495281 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495286 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495290 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.497009 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495297 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495304 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495309 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495314 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495319 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495323 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495327 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495332 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495336 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495340 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495346 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495351 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495355 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495359 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495363 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495367 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495371 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495376 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495380 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495384 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.497821 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495388 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495393 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495397 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495401 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495405 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495409 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495414 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495418 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495422 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495426 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495430 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495435 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495439 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495443 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495447 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495451 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495456 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495460 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495464 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495469 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.498684 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495474 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495481 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495485 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.495491 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496861 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496872 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496876 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496881 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496885 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496890 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496895 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496899 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496903 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496908 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496912 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496916 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496920 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496924 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496929 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.499375 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496933 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496938 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496942 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496946 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496950 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496954 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496959 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496963 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496967 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496971 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496975 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496979 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496984 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496989 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496993 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.496998 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497002 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497007 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497011 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.499869 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497015 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497019 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497024 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497027 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497032 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497036 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497040 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497044 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497049 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497054 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497061 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497067 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497071 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497077 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497082 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497088 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497093 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497098 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497104 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497108 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.500445 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497112 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497116 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497120 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497124 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497128 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497133 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497138 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497143 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497147 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497152 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497156 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497162 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497167 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497171 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497176 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497180 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497184 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497189 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497193 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497198 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.500949 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497202 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497206 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497210 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497214 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497218 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497223 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497227 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497231 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497236 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497240 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497244 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497248 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497252 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497257 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497261 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497266 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497270 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497274 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497279 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.501463 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497283 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497288 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497292 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497296 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497301 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497307 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497314 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497319 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497327 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497332 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497336 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497340 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.497345 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498225 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498237 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498242 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498247 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498252 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498256 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.501954 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498261 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498265 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498269 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498274 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498278 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498282 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498286 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498290 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498295 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498299 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498303 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498308 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498313 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498318 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498322 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498326 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498330 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498334 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498340 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498344 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.502419 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498348 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498352 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498356 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498360 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498364 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498368 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498372 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498377 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498381 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498386 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498390 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498394 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498398 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498403 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498407 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498411 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498415 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498420 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498424 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.502933 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498430 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498434 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498438 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498442 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498447 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498451 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498456 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498460 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498464 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498468 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498473 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498477 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498486 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498490 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498494 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498500 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498506 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498511 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498515 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.503388 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498519 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498523 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498527 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498531 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498535 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498539 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498543 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498547 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498552 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498556 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498560 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498565 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498569 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498573 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498577 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498581 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498585 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498590 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498595 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498599 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.503866 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498603 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498627 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498632 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498636 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498640 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498644 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498649 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498653 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498657 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498661 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498665 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498669 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498673 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498678 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498682 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498685 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498689 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498693 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498697 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498701 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.504345 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498705 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.498709 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498861 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498874 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498886 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498893 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498900 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498905 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498912 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498919 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498925 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498930 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498937 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498942 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498947 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498953 2575 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498957 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498963 2575 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498967 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498973 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498979 2575 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498984 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498989 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498994 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:13:35.504848 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.498999 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499006 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499011 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499016 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499022 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499027 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499032 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499037 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499042 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499047 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499053 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499058 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499063 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499068 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499073 2575 flags.go:64] FLAG: --enable-server="true" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499078 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499086 2575 flags.go:64] FLAG: --event-burst="100" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499091 2575 flags.go:64] FLAG: --event-qps="50" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499096 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499102 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499107 2575 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499115 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499120 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499125 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499130 2575 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:13:35.505423 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499135 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499140 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499144 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499150 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499155 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499160 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499165 2575 flags.go:64] FLAG: --feature-gates="" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499171 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499176 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499220 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499237 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499242 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499247 2575 flags.go:64] FLAG: --help="false" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499251 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499255 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499261 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499267 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499273 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499279 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499282 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499285 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499289 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499292 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:13:35.506089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499295 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499298 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499301 2575 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499305 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499308 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499311 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499314 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499317 2575 flags.go:64] FLAG: --lock-file="" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499320 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499323 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499326 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499338 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499343 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499349 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499354 2575 flags.go:64] FLAG: --logging-format="text" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499359 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499362 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499365 2575 flags.go:64] FLAG: --manifest-url="" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499368 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499381 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499385 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499389 2575 flags.go:64] FLAG: --max-pods="110" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499392 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499395 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499399 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:13:35.506676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499402 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499405 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499408 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499411 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499420 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499425 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499431 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499437 2575 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499441 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499444 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499447 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499450 2575 flags.go:64] FLAG: --port="10250" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499453 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499456 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00356a433a01daf35" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499459 2575 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499462 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499465 2575 flags.go:64] FLAG: --register-node="true" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499468 2575 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499472 2575 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499475 2575 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499478 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499481 2575 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499485 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499488 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499491 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499494 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:13:35.507304 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499499 2575 flags.go:64] FLAG: --runonce="false" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499509 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499516 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499520 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499525 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499528 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499531 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499534 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499537 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499540 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499543 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499546 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499555 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499559 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499562 2575 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499565 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499574 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499577 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499580 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499590 2575 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499595 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499600 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499605 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499624 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499628 2575 flags.go:64] FLAG: --v="2" Nov 03 16:13:35.508247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499633 2575 flags.go:64] FLAG: --version="false" Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499637 2575 flags.go:64] FLAG: --vmodule="" Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499641 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.499645 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499890 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499896 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499899 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499905 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499907 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499911 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499913 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499917 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499921 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499926 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499933 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499939 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499943 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499945 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499948 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499951 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.509040 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499953 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499956 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499958 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499961 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499963 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499966 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499968 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499971 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499974 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499977 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499980 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499982 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499985 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499988 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499990 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499993 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499995 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.499998 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500001 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500008 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.509526 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500012 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500017 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500021 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500023 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500026 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500029 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500032 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500034 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500037 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500040 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500042 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500045 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500048 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500050 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500053 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500055 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500057 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500060 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500063 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.510035 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500065 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500068 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500070 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500074 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500077 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500080 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500085 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500089 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500094 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500098 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500101 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500104 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500108 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500111 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500113 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500116 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500119 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500122 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500124 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500127 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.510485 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500130 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500132 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500135 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500137 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500140 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500142 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500145 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500147 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500150 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500152 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500163 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500168 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500173 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500177 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500182 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500187 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500189 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500580 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500587 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.511016 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500590 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500594 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500598 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500603 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500607 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500629 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500633 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500636 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500640 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500644 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500648 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500652 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.500693 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500734 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500740 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500743 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.511475 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500745 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500748 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500750 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500753 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500757 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500761 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500766 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500770 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500774 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500777 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500779 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500782 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500784 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500788 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500791 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500793 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500796 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500799 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500802 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.511888 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500804 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500807 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500811 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500813 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500816 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500819 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500822 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500825 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500829 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500832 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500835 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500839 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500844 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500848 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500853 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500856 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500859 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500861 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500864 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500866 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.512358 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500869 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500873 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500876 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500879 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500881 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500884 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500889 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500891 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500894 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500896 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500899 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500901 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500904 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500906 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500910 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500913 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500916 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500918 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500923 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.512879 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500928 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500933 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500939 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500942 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500944 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500946 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500949 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500951 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500954 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500957 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500960 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500962 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500965 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500968 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500970 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500972 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500975 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500978 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500980 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.513319 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500984 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500987 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500989 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500992 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500994 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500996 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.500999 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501003 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501009 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501014 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501018 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501022 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501025 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501027 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501030 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501032 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501035 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501038 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501040 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501043 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.513784 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501048 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501050 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501059 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501061 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.501064 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.510572 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.510597 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.511581 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.511588 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511679 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511686 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511691 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511694 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511697 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511700 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511703 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511705 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511708 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511711 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.514246 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511714 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511717 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511720 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511723 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511725 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511728 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511730 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511733 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511735 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511738 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511740 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511759 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511763 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511765 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511768 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511771 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511773 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511776 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511778 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.514744 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511781 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511783 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511786 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511788 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511791 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511794 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511796 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511799 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511802 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511804 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511807 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511809 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511812 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511815 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511817 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511820 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511823 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511826 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511828 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.515191 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511831 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511833 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511836 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511838 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511841 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511843 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511846 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511848 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511851 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511854 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511856 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511858 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511861 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511864 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511866 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511869 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511873 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511878 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511881 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.515640 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511884 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511887 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511889 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511892 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511895 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511898 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511900 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511903 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511906 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511908 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511911 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511913 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511917 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511919 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511922 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511924 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511927 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511929 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511932 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511934 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.516114 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511937 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511939 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511943 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511945 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511948 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511951 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511953 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511956 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511958 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511961 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511963 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511966 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511969 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511971 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511975 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511977 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511979 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511982 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.516589 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.511986 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.512021 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512207 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512213 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512217 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512220 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512223 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512226 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512229 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512232 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512235 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512238 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512240 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512243 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512245 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512248 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.517036 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512250 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512253 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512255 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512258 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512260 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512262 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512265 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512268 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512271 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512273 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512277 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512279 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512282 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512285 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512287 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512290 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512292 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512295 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512298 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.517403 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512300 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512302 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512305 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512308 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512310 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512313 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512316 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512318 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512321 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512323 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512326 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512328 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512333 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512337 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512340 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512343 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512345 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512348 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512350 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.517855 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512353 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512356 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512358 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512361 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512364 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512366 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512369 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512372 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512374 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512377 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512379 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512382 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512385 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512387 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512389 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512392 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512394 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512397 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512399 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512402 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.518284 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512405 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512407 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512410 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512414 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512417 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512420 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512422 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512425 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512427 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512430 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512432 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512434 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512437 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512439 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512442 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512444 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512447 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512449 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512452 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.518771 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512455 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512457 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512459 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512462 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512465 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512468 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512471 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512473 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512475 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512478 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512480 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512482 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512485 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512487 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512490 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.512515 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:35.519211 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512575 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512579 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512582 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512585 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512588 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512591 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512593 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512596 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512599 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512602 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512604 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512607 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512628 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512632 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512635 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512638 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512642 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512644 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512647 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.519588 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512649 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512653 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512655 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512658 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512660 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512662 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512665 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512668 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512670 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512672 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512675 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512678 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512681 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512683 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512686 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512688 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512691 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512693 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512696 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512698 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.520086 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512701 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512704 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512706 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512710 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512713 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512716 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512719 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512722 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512724 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512727 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512730 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512735 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512737 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512740 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512742 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512744 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512747 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512749 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.520566 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512752 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512754 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512757 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512759 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512762 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512766 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512768 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512771 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512773 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512776 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512778 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512783 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512786 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512788 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512790 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512793 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512795 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512798 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512800 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512803 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.521156 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512805 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512808 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512811 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512813 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512816 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512818 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512822 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512824 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512827 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512829 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512832 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512834 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512837 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512839 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512842 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512846 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512848 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512850 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512854 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.521648 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512856 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512859 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512861 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512864 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512867 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512870 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512872 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512875 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:35.512878 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.512950 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.515598 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.516651 2575 server.go:1014] "Starting client certificate rotation" Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.516744 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.518382 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.518911 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:13:35.522085 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.520099 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:13:35.534366 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.534336 2575 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:13:35.537573 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.537556 2575 log.go:25] "Validated CRI v1 image API" Nov 03 16:13:35.538490 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.538458 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:13:35.543057 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.543022 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a33cdbbe-5cde-46e8-a6a5-fa8a37d32586:/dev/nvme0n1p3 d409c9a5-33f4-40f1-8efd-218884bf3e1e:/dev/nvme0n1p4] Nov 03 16:13:35.543155 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.543052 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 16:13:35.551243 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.551075 2575 manager.go:217] Machine: {Timestamp:2026-11-03 16:13:35.549523768 +0000 UTC m=+0.375608778 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098798 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23724e6c73b018319dbeb90c4ce9a8 SystemUUID:ec23724e-6c73-b018-319d-beb90c4ce9a8 BootID:6b66123d-8682-4e82-83e8-1b8b25b10719 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:ae:3c:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:ae:3c:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:73:b6:51:ee:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:13:35.551968 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.551953 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:13:35.552138 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.552122 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:13:35.555009 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.554964 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:13:35.555166 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.555012 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:13:35.555215 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.555170 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:13:35.555215 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.555177 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:13:35.555268 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.555224 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:13:35.555268 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.555245 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:13:35.556509 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.556497 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:13:35.561215 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.561196 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:13:35.561388 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.561378 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:13:35.562216 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.562200 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:13:35.562255 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.562221 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:13:35.562255 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.562232 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:13:35.562255 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.562241 2575 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:13:35.562255 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.562250 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:13:35.562746 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.562729 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:13:35.566214 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.566193 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:13:35.572875 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.572862 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:13:35.573797 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.573785 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:13:35.573834 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.573815 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:13:35.573971 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.573960 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:13:35.574003 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.573980 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:13:35.574003 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.573991 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:13:35.574003 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574000 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:13:35.574080 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574010 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:13:35.574080 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574018 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:13:35.574919 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574688 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:13:35.574919 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574695 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:13:35.574919 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574704 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:13:35.574919 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574709 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:13:35.574919 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574718 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:13:35.574919 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.574728 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:13:35.575441 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.575432 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:13:35.575470 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.575445 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:13:35.579545 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.579534 2575 server.go:1263] "Started kubelet" Nov 03 16:13:35.579697 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.579653 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:13:35.579765 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.579667 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:13:35.579765 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.579722 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:13:35.580376 ip-10-0-139-190 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:13:35.581243 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.581221 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:13:35.582771 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.582752 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:13:35.587338 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.587310 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 16:13:35.590124 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.590104 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:13:35.590749 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.590731 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:13:35.590837 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.590820 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found" Nov 03 16:13:35.591525 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591506 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:13:35.591525 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591527 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:13:35.591690 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591634 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:13:35.591690 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591648 2575 factory.go:55] Registering systemd factory Nov 03 16:13:35.591690 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591653 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:13:35.591690 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591657 2575 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:13:35.591867 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591710 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:13:35.591867 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.591717 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:13:35.592066 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.592050 2575 factory.go:153] Registering CRI-O factory Nov 03 16:13:35.592101 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.592070 2575 factory.go:223] Registration of the crio container factory successfully Nov 03 16:13:35.592101 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.592095 2575 factory.go:103] Registering Raw factory Nov 03 16:13:35.592168 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.592126 2575 manager.go:1199] Started watching for new ooms in manager Nov 03 16:13:35.593254 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.593236 2575 manager.go:319] Starting recovery of all containers Nov 03 16:13:35.602687 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.602663 2575 manager.go:324] Recovery completed Nov 03 16:13:35.606923 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.606906 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.608756 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.608714 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.608837 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.608782 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.608837 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.608795 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.609345 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.609331 2575 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:13:35.609345 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.609344 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:13:35.609436 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.609360 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:13:35.609436 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.609335 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.610107 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.610091 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.610164 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.610119 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.610164 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.610137 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.611692 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.611680 2575 policy_none.go:50] "Start" Nov 03 16:13:35.611726 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.611698 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:13:35.611726 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.611711 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:13:35.613738 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.613727 2575 policy_none.go:44] "Start" Nov 03 16:13:35.616129 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.616113 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:13:35.616224 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.616152 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49wm5" Nov 03 16:13:35.617210 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.617188 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 16:13:35.617294 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.617189 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 16:13:35.618382 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.617194 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-190.ec2.internal.18e495088e45a526 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-190.ec2.internal,},FirstTimestamp:2026-11-03 16:13:35.579505958 +0000 UTC m=+0.405590968,LastTimestamp:2026-11-03 16:13:35.579505958 +0000 UTC m=+0.405590968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-190.ec2.internal,}" Nov 03 16:13:35.628383 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.628361 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49wm5" Nov 03 16:13:35.640721 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.640646 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-190.ec2.internal.18e495089003fcb5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-190.ec2.internal,},FirstTimestamp:2026-11-03 16:13:35.608757429 +0000 UTC m=+0.434842453,LastTimestamp:2026-11-03 16:13:35.608757429 +0000 UTC m=+0.434842453,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-190.ec2.internal,}" Nov 03 16:13:35.641553 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.641529 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.657000 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.659412 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.663779 2575 manager.go:341] "Starting Device Plugin manager" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.663807 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.663816 2575 server.go:87] "Starting device plugin registration server" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.664047 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.664059 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.664210 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.664314 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.664323 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.665037 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.665085 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-190.ec2.internal\" not found" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.668386 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.671911 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.673299 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.673321 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.673340 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:13:35.675747 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.673369 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 16:13:35.689505 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.689489 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:35.764856 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.764789 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.765749 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.765722 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.765861 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.765758 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.765861 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.765771 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.765861 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.765801 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.773954 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.773933 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"] Nov 03 16:13:35.773999 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.773990 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.774756 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.774741 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.774813 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.774768 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.774813 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.774783 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.776983 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.776971 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.777131 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.777117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.777166 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.777149 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.777970 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.777953 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.778060 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.777986 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.778060 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.777997 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.778060 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.777958 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.778060 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.778057 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.778216 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.778070 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.780128 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.780113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.780219 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.780137 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.780827 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.780808 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.780909 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.780844 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.780909 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.780861 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.786741 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.786726 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.786824 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.786750 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-190.ec2.internal\": node \"ip-10-0-139-190.ec2.internal\" not found" Nov 03 16:13:35.792097 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.792080 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.794324 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.792208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.794324 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.792229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.794324 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.792245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f77174fff7f21e40f2a31f54c08055a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"9f77174fff7f21e40f2a31f54c08055a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.815398 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.815383 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.817867 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.817851 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.840281 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.840266 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:35.845742 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.845726 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:35.856951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.856936 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:35.857033 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.856973 2575 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.857033 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.856984 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.891010 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.890991 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:35.891083 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:35.891027 2575 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.893179 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.893167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.893227 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.893186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.893227 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.893204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f77174fff7f21e40f2a31f54c08055a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"9f77174fff7f21e40f2a31f54c08055a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.893296 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.893249 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f77174fff7f21e40f2a31f54c08055a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"9f77174fff7f21e40f2a31f54c08055a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.893296 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.893284 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:35.893363 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:35.893284 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:36.146636 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.146547 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:13:36.150318 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.150298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:13:36.517640 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.517550 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 16:13:36.518098 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.517718 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:13:36.562916 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.562891 2575 apiserver.go:52] "Watching apiserver" Nov 03 16:13:36.590567 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.590542 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 16:13:36.604294 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.604276 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:36.610230 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.610212 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:13:36.610608 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.610593 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal","openshift-multus/multus-w6prk","openshift-network-diagnostics/network-check-target-p48p4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt","openshift-cluster-node-tuning-operator/tuned-f9qd4","kube-system/global-pull-secret-syncer-h6g2k","openshift-multus/multus-additional-cni-plugins-6x8qs","openshift-ovn-kubernetes/ovnkube-node-gl2kp","openshift-dns/node-resolver-7dwpd","openshift-image-registry/node-ca-9wmmj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal","openshift-multus/network-metrics-daemon-wfft8","openshift-network-operator/iptables-alerter-fvnv7","kube-system/konnectivity-agent-bvp5p"] Nov 03 16:13:36.612893 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.612875 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.616008 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.615500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:36.616008 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.615580 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:36.617388 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.617369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.620276 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.620258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.622446 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.622432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.622517 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.622488 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:36.624696 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.624670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.627088 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.627068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.628430 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.628411 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n246q\"" Nov 03 16:13:36.629451 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.629429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.631389 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.631373 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:13:36.631587 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.631576 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.631764 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.631748 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:13:36.631820 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.631764 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.631903 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.631890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.633867 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.633831 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:08:36 +0000 UTC" deadline="2028-06-05 19:41:45.211963655 +0000 UTC" Nov 03 16:13:36.633867 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.633866 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13923h28m8.57809983s" Nov 03 16:13:36.634104 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.634094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:36.634162 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.634148 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:36.635724 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.635707 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:13:36.636437 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.636413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.638779 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.638758 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.640087 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.640072 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:13:36.641539 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.641516 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4w5fx" Nov 03 16:13:36.657633 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.657601 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4w5fx" Nov 03 16:13:36.677868 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:36.677835 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f77174fff7f21e40f2a31f54c08055a.slice/crio-d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb WatchSource:0}: Error finding container d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb: Status 404 returned error can't find the container with id d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb Nov 03 16:13:36.678096 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:36.678081 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e318b374c3998927ea8465c3974849.slice/crio-5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45 WatchSource:0}: Error finding container 5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45: Status 404 returned error can't find the container with id 5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45 Nov 03 16:13:36.682334 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.682320 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:13:36.693168 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.693149 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:13:36.697310 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-socket-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.697402 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-node-log\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.697402 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-konnectivity-ca\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.697471 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovn-node-metrics-cert\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.697471 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-conf-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.697471 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-kubelet\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.697591 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysconfig\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.697591 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-systemd\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.697695 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-tmp-dir\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.697695 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cnibin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.697787 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-hostroot\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.697787 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.697787 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-hosts-file\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.697888 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ddfba7-7752-4fad-be36-3f8d29b64a2b-iptables-alerter-script\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.697888 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-var-lib-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.697888 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw96t\" (UniqueName: \"kubernetes.io/projected/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-kube-api-access-pw96t\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.697990 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-sys-fs\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.697990 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.697990 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-bin\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698140 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.697997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698140 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-script-lib\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698140 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-netns\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.698140 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-multus-certs\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.698140 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-host\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.698337 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjpt\" (UniqueName: \"kubernetes.io/projected/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kube-api-access-kvjpt\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.698337 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.698337 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cnibin\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.698337 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.698337 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.698337 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-ovn\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698582 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-log-socket\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698582 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.698582 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-config\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698582 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5a6779b-623f-4618-bc63-f9efc8008fa4-serviceca\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.698582 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-modprobe-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.698582 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-tuned\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-system-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-k8s-cni-cncf-io\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-registration-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-device-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:36.698876 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-agent-certs\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-kubernetes\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.698991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5988q\" (UniqueName: \"kubernetes.io/projected/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-kube-api-access-5988q\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-systemd-units\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5a6779b-623f-4618-bc63-f9efc8008fa4-host\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-var-lib-kubelet\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-bin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.699151 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-daemon-config\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.699458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-netns\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.699458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-etc-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.699458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.699458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-env-overrides\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.699458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-conf\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.699458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-lib-modules\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.699736 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.699736 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cs9\" (UniqueName: \"kubernetes.io/projected/580cbd81-dec7-4b31-ab6c-e660173aecb4-kube-api-access-88cs9\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.699736 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ddfba7-7752-4fad-be36-3f8d29b64a2b-host-slash\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.699736 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshsf\" (UniqueName: \"kubernetes.io/projected/25ddfba7-7752-4fad-be36-3f8d29b64a2b-kube-api-access-vshsf\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.699736 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-socket-dir-parent\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.699956 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6xr\" (UniqueName: \"kubernetes.io/projected/174d66ce-30d8-4470-91e1-058c1e7e0c3b-kube-api-access-cj6xr\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.699956 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-tmp\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.699956 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cni-binary-copy\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.699956 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-kubelet\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.699956 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-etc-kubernetes\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.700189 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.699949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjhl\" (UniqueName: \"kubernetes.io/projected/d5a6779b-623f-4618-bc63-f9efc8008fa4-kube-api-access-vbjhl\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.700189 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwbx\" (UniqueName: \"kubernetes.io/projected/fdc671ec-9c42-41b8-983f-9385a62b14ba-kube-api-access-9xwbx\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:36.700189 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-systemd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.700628 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-netd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.700688 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-sys\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.700746 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.700746 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tx5h\" (UniqueName: \"kubernetes.io/projected/8d31c32d-0e82-44a9-b9f1-e38073708fe6-kube-api-access-5tx5h\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.700831 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.700831 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-run\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.700831 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.700932 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-slash\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.700932 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-kubelet-config\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.701010 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-dbus\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.701010 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.700951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-multus\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.728789 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.728764 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlmt\"" Nov 03 16:13:36.728889 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.728814 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9tss\"" Nov 03 16:13:36.728972 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.728957 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tf624\"" Nov 03 16:13:36.729103 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729090 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5xwqx\"" Nov 03 16:13:36.729166 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729152 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tn5d9\"" Nov 03 16:13:36.729166 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729159 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:13:36.729265 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729156 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6q76v\"" Nov 03 16:13:36.729265 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729180 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:13:36.729265 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729197 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hxhwt\"" Nov 03 16:13:36.729265 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.729204 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2k4df\"" Nov 03 16:13:36.732561 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732548 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.732644 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732631 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.732704 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732645 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.732704 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732648 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.732704 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732667 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.732704 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732681 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732724 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732731 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732752 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732758 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732816 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732818 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:13:36.732895 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.732828 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.733150 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.733016 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.741467 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.741449 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:13:36.741548 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.741491 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:13:36.742059 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.742042 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:13:36.742137 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.742071 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:13:36.742137 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.742089 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:13:36.742137 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.742125 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:13:36.801701 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-lib-modules\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.801701 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.801701 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cs9\" (UniqueName: \"kubernetes.io/projected/580cbd81-dec7-4b31-ab6c-e660173aecb4-kube-api-access-88cs9\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ddfba7-7752-4fad-be36-3f8d29b64a2b-host-slash\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshsf\" (UniqueName: \"kubernetes.io/projected/25ddfba7-7752-4fad-be36-3f8d29b64a2b-kube-api-access-vshsf\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-socket-dir-parent\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6xr\" (UniqueName: \"kubernetes.io/projected/174d66ce-30d8-4470-91e1-058c1e7e0c3b-kube-api-access-cj6xr\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-tmp\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cni-binary-copy\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801826 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ddfba7-7752-4fad-be36-3f8d29b64a2b-host-slash\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801844 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-lib-modules\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-kubelet\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-kubelet\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-etc-kubernetes\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-etc-kubernetes\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801931 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-socket-dir-parent\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.801951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.801937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjhl\" (UniqueName: \"kubernetes.io/projected/d5a6779b-623f-4618-bc63-f9efc8008fa4-kube-api-access-vbjhl\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802111 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwbx\" (UniqueName: \"kubernetes.io/projected/fdc671ec-9c42-41b8-983f-9385a62b14ba-kube-api-access-9xwbx\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-systemd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-netd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-sys\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-systemd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tx5h\" (UniqueName: \"kubernetes.io/projected/8d31c32d-0e82-44a9-b9f1-e38073708fe6-kube-api-access-5tx5h\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802242 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-netd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-run\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802306 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-slash\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-kubelet-config\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802375 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-slash\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-dbus\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.802570 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-run\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-multus\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802424 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-socket-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802448 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-kubelet-config\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-multus\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-node-log\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802467 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-dbus\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-konnectivity-ca\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802501 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-node-log\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802516 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-sys\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovn-node-metrics-cert\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-conf-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-kubelet\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802698 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-conf-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysconfig\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-systemd\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.803390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-tmp-dir\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cnibin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-hostroot\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802853 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-systemd\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysconfig\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-hostroot\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802892 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cnibin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-kubelet\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-hosts-file\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ddfba7-7752-4fad-be36-3f8d29b64a2b-iptables-alerter-script\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.802995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-var-lib-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-hosts-file\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw96t\" (UniqueName: \"kubernetes.io/projected/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-kube-api-access-pw96t\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.803041 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-sys-fs\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.804239 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.803120 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret podName:e0165af3-4e86-4907-8919-6ec892f905b7 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:37.3030845 +0000 UTC m=+2.129169520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret") pod "global-pull-secret-syncer-h6g2k" (UID: "e0165af3-4e86-4907-8919-6ec892f905b7") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-bin\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-script-lib\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-netns\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-multus-certs\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-host\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjpt\" (UniqueName: \"kubernetes.io/projected/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kube-api-access-kvjpt\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cnibin\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-ovn\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-log-socket\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803442 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-var-lib-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-config\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805115 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5a6779b-623f-4618-bc63-f9efc8008fa4-serviceca\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-modprobe-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-tuned\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-system-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-k8s-cni-cncf-io\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803645 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-tmp-dir\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-registration-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-device-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803729 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-socket-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-agent-certs\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-ovn\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-kubernetes\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-bin\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5988q\" (UniqueName: \"kubernetes.io/projected/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-kube-api-access-5988q\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.805907 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-systemd-units\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803919 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-konnectivity-ca\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5a6779b-623f-4618-bc63-f9efc8008fa4-host\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803942 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.803986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-var-lib-kubelet\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804042 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-sys-fs\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804041 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-k8s-cni-cncf-io\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.804119 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804123 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-kubernetes\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804145 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.804175 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:13:37.304150703 +0000 UTC m=+2.130235716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804193 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-device-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804222 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ddfba7-7752-4fad-be36-3f8d29b64a2b-iptables-alerter-script\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804312 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-multus-certs\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804320 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-host\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.806429 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-bin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-daemon-config\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-netns\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-etc-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-env-overrides\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-conf\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-registration-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-conf\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804760 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-config\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-script-lib\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804785 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-bin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-systemd-units\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.804985 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cnibin\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805051 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-system-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805088 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-log-socket\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807102 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805093 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-modprobe-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-netns\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805230 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5a6779b-623f-4618-bc63-f9efc8008fa4-serviceca\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805289 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5a6779b-623f-4618-bc63-f9efc8008fa4-host\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805335 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805379 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-var-lib-kubelet\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-etc-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805380 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-netns\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-daemon-config\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805747 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-env-overrides\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.805765 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cni-binary-copy\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.806066 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-tmp\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.806151 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovn-node-metrics-cert\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.806906 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-agent-certs\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:36.807568 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.807557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-tuned\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.826046 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.826020 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshsf\" (UniqueName: \"kubernetes.io/projected/25ddfba7-7752-4fad-be36-3f8d29b64a2b-kube-api-access-vshsf\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:36.826203 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.826174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjhl\" (UniqueName: \"kubernetes.io/projected/d5a6779b-623f-4618-bc63-f9efc8008fa4-kube-api-access-vbjhl\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:36.826487 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.826468 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cs9\" (UniqueName: \"kubernetes.io/projected/580cbd81-dec7-4b31-ab6c-e660173aecb4-kube-api-access-88cs9\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.826705 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.826686 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tx5h\" (UniqueName: \"kubernetes.io/projected/8d31c32d-0e82-44a9-b9f1-e38073708fe6-kube-api-access-5tx5h\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:36.826743 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.826691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6xr\" (UniqueName: \"kubernetes.io/projected/174d66ce-30d8-4470-91e1-058c1e7e0c3b-kube-api-access-cj6xr\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.827299 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.827274 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwbx\" (UniqueName: \"kubernetes.io/projected/fdc671ec-9c42-41b8-983f-9385a62b14ba-kube-api-access-9xwbx\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:36.832480 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.832458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw96t\" (UniqueName: \"kubernetes.io/projected/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-kube-api-access-pw96t\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:36.833386 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.833370 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:36.833466 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.833392 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:36.833466 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.833405 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lc4lw for pod openshift-network-diagnostics/network-check-target-p48p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:36.833466 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:36.833460 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw podName:3f8aeec1-65f6-482a-b42d-873c5e7c1903 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:37.333441928 +0000 UTC m=+2.159526943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lc4lw" (UniqueName: "kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw") pod "network-check-target-p48p4" (UID: "3f8aeec1-65f6-482a-b42d-873c5e7c1903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:36.835532 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.835507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5988q\" (UniqueName: \"kubernetes.io/projected/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-kube-api-access-5988q\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:36.835608 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.835594 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjpt\" (UniqueName: \"kubernetes.io/projected/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kube-api-access-kvjpt\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.945604 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.945571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w6prk" Nov 03 16:13:36.955342 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:36.955318 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174d66ce_30d8_4470_91e1_058c1e7e0c3b.slice/crio-860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda WatchSource:0}: Error finding container 860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda: Status 404 returned error can't find the container with id 860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda Nov 03 16:13:36.965196 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.965176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:13:36.974539 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:36.974504 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8cc36e_da75_4685_88ae_5fa8db05b7a3.slice/crio-57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a WatchSource:0}: Error finding container 57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a: Status 404 returned error can't find the container with id 57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a Nov 03 16:13:36.990269 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:36.990243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:13:36.998599 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:36.998574 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580cbd81_dec7_4b31_ab6c_e660173aecb4.slice/crio-c3879811a29c19d99a2a645cba7c58682b202f1677157c9507cfb2c95dd814ca WatchSource:0}: Error finding container c3879811a29c19d99a2a645cba7c58682b202f1677157c9507cfb2c95dd814ca: Status 404 returned error can't find the container with id c3879811a29c19d99a2a645cba7c58682b202f1677157c9507cfb2c95dd814ca Nov 03 16:13:37.009185 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.009168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:13:37.013934 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.013917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:13:37.016062 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.016044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:13:37.017289 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:37.017268 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac12a2a_91d1_4812_bf79_8d4ab8bb4a8c.slice/crio-d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8 WatchSource:0}: Error finding container d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8: Status 404 returned error can't find the container with id d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8 Nov 03 16:13:37.024194 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.024173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:13:37.024420 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:37.024404 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d31c32d_0e82_44a9_b9f1_e38073708fe6.slice/crio-4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec WatchSource:0}: Error finding container 4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec: Status 404 returned error can't find the container with id 4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec Nov 03 16:13:37.025032 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:37.025011 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a6779b_623f_4618_bc63_f9efc8008fa4.slice/crio-027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557 WatchSource:0}: Error finding container 027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557: Status 404 returned error can't find the container with id 027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557 Nov 03 16:13:37.025716 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.025698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:37.029536 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.029515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:37.036416 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:37.036379 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ddfba7_7752_4fad_be36_3f8d29b64a2b.slice/crio-ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7 WatchSource:0}: Error finding container ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7: Status 404 returned error can't find the container with id ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7 Nov 03 16:13:37.040387 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:13:37.040355 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf7557b_7e3c_49a9_a522_1e46ba83b79f.slice/crio-e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af WatchSource:0}: Error finding container e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af: Status 404 returned error can't find the container with id e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af Nov 03 16:13:37.149847 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.149780 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:37.308318 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.308287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:37.308471 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.308332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:37.308471 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.308458 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:37.308546 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.308506 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:13:38.308494276 +0000 UTC m=+3.134579288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:37.308596 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.308567 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:37.308596 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.308588 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret podName:e0165af3-4e86-4907-8919-6ec892f905b7 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:38.308581999 +0000 UTC m=+3.134667011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret") pod "global-pull-secret-syncer-h6g2k" (UID: "e0165af3-4e86-4907-8919-6ec892f905b7") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:37.409321 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:37.408671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:37.409321 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.408856 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:37.409321 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.408875 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:37.409321 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.408887 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lc4lw for pod openshift-network-diagnostics/network-check-target-p48p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:37.409321 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:37.408942 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw podName:3f8aeec1-65f6-482a-b42d-873c5e7c1903 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:38.408925606 +0000 UTC m=+3.235010625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lc4lw" (UniqueName: "kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw") pod "network-check-target-p48p4" (UID: "3f8aeec1-65f6-482a-b42d-873c5e7c1903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:39.024075 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.023967 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:38 +0000 UTC" deadline="2028-06-18 01:07:30.756831172 +0000 UTC" Nov 03 16:13:39.024075 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.024028 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14216h53m51.732806857s" Nov 03 16:13:39.038955 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.038730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:39.038955 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.038855 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:39.069377 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.069319 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bvp5p" event={"ID":"8bf7557b-7e3c-49a9-a522-1e46ba83b79f","Type":"ContainerStarted","Data":"e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af"} Nov 03 16:13:39.078641 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.078610 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fvnv7" event={"ID":"25ddfba7-7752-4fad-be36-3f8d29b64a2b","Type":"ContainerStarted","Data":"ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7"} Nov 03 16:13:39.089503 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.089476 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerStarted","Data":"4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec"} Nov 03 16:13:39.097329 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.097285 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" event={"ID":"580cbd81-dec7-4b31-ab6c-e660173aecb4","Type":"ContainerStarted","Data":"c3879811a29c19d99a2a645cba7c58682b202f1677157c9507cfb2c95dd814ca"} Nov 03 16:13:39.111869 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.111834 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" event={"ID":"fe8cc36e-da75-4685-88ae-5fa8db05b7a3","Type":"ContainerStarted","Data":"57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a"} Nov 03 16:13:39.125647 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.125607 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w6prk" event={"ID":"174d66ce-30d8-4470-91e1-058c1e7e0c3b","Type":"ContainerStarted","Data":"860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda"} Nov 03 16:13:39.148503 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.148221 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" event={"ID":"9f77174fff7f21e40f2a31f54c08055a","Type":"ContainerStarted","Data":"d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb"} Nov 03 16:13:39.174769 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.174741 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wmmj" event={"ID":"d5a6779b-623f-4618-bc63-f9efc8008fa4","Type":"ContainerStarted","Data":"027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557"} Nov 03 16:13:39.182252 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.182230 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:39.188416 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.188390 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7dwpd" event={"ID":"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c","Type":"ContainerStarted","Data":"d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8"} Nov 03 16:13:39.189760 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.189725 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"c7e318b374c3998927ea8465c3974849","Type":"ContainerStarted","Data":"5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45"} Nov 03 16:13:39.204371 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.204344 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"05291fcf449d86065117ef2e707aadb11d15b311a87194fec51667682a7e89eb"} Nov 03 16:13:39.677664 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.677629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:39.677854 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.677723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:39.677854 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.677834 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:39.677964 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.677868 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:39.677964 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.677908 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:13:41.677888968 +0000 UTC m=+5.141163369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:39.677964 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.677928 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret podName:e0165af3-4e86-4907-8919-6ec892f905b7 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:41.677919188 +0000 UTC m=+5.141193591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret") pod "global-pull-secret-syncer-h6g2k" (UID: "e0165af3-4e86-4907-8919-6ec892f905b7") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:39.778671 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:39.778627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:39.778868 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.778850 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:39.778932 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.778876 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:39.778932 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.778890 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lc4lw for pod openshift-network-diagnostics/network-check-target-p48p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:39.779035 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:39.778943 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw podName:3f8aeec1-65f6-482a-b42d-873c5e7c1903 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:41.778926517 +0000 UTC m=+5.242200922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lc4lw" (UniqueName: "kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw") pod "network-check-target-p48p4" (UID: "3f8aeec1-65f6-482a-b42d-873c5e7c1903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:40.024635 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:40.024549 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:38 +0000 UTC" deadline="2028-06-16 05:17:35.462477199 +0000 UTC" Nov 03 16:13:40.024635 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:40.024590 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14173h3m55.43789177s" Nov 03 16:13:40.037170 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:40.037142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:40.037311 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:40.037280 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:40.037697 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:40.037678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:40.037800 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:40.037778 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:41.037951 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:41.037687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:41.037951 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.037841 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:41.698578 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:41.698534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:41.698729 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:41.698621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:41.698811 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.698792 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:41.698902 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.698891 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:13:45.698869645 +0000 UTC m=+9.162144045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:41.699345 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.699325 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:41.699408 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.699388 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret podName:e0165af3-4e86-4907-8919-6ec892f905b7 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:45.699372657 +0000 UTC m=+9.162647058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret") pod "global-pull-secret-syncer-h6g2k" (UID: "e0165af3-4e86-4907-8919-6ec892f905b7") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:41.799451 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:41.799245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:41.799613 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.799590 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:41.799613 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.799610 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:41.799728 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.799624 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lc4lw for pod openshift-network-diagnostics/network-check-target-p48p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:41.799728 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:41.799670 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw podName:3f8aeec1-65f6-482a-b42d-873c5e7c1903 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:45.799652277 +0000 UTC m=+9.262926706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lc4lw" (UniqueName: "kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw") pod "network-check-target-p48p4" (UID: "3f8aeec1-65f6-482a-b42d-873c5e7c1903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:42.037183 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.037151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:42.037354 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:42.037279 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:42.037689 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.037670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:42.037784 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:42.037765 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:42.227301 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.227228 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="00481ae7c3c5f47c67dd68ed7b0c16264e94b8bd8bdb5c6a55aed3a9d63b2d35" exitCode=0 Nov 03 16:13:42.227707 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.227325 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerDied","Data":"00481ae7c3c5f47c67dd68ed7b0c16264e94b8bd8bdb5c6a55aed3a9d63b2d35"} Nov 03 16:13:42.243158 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.243122 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" event={"ID":"9f77174fff7f21e40f2a31f54c08055a","Type":"ContainerStarted","Data":"3e178774db051d43e8ba27c39a8e4d11836cdcbba47693ba43c3c9207c21b159"} Nov 03 16:13:42.250554 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.250526 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7e318b374c3998927ea8465c3974849" containerID="991272ca83580ae41accb891c986bad156d28d7e153d71eaaaab56b73138566b" exitCode=0 Nov 03 16:13:42.250657 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.250568 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"c7e318b374c3998927ea8465c3974849","Type":"ContainerDied","Data":"991272ca83580ae41accb891c986bad156d28d7e153d71eaaaab56b73138566b"} Nov 03 16:13:42.338356 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:42.338216 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" podStartSLOduration=5.338198298 podStartE2EDuration="5.338198298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:42.337682222 +0000 UTC m=+5.800956643" watchObservedRunningTime="2026-11-03 16:13:42.338198298 +0000 UTC m=+5.801472718" Nov 03 16:13:43.043107 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:43.043068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:43.043288 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:43.043206 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:43.265172 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:43.265087 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bvp5p" event={"ID":"8bf7557b-7e3c-49a9-a522-1e46ba83b79f","Type":"ContainerStarted","Data":"baa66fc1c181b0e9897af94acffcbb72693b309c1db0c6385da7724668619dda"} Nov 03 16:13:43.271061 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:43.271017 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"c7e318b374c3998927ea8465c3974849","Type":"ContainerStarted","Data":"50c8820ee3f80869f09f65f33788bbbd343612923e31d15db72d013d5152b462"} Nov 03 16:13:43.343589 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:43.343529 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" podStartSLOduration=6.343514237 podStartE2EDuration="6.343514237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:43.343335525 +0000 UTC m=+6.806609944" watchObservedRunningTime="2026-11-03 16:13:43.343514237 +0000 UTC m=+6.806788659" Nov 03 16:13:43.343762 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:43.343681 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bvp5p" podStartSLOduration=2.071127417 podStartE2EDuration="6.343674416s" totalImagesPullingTime="4.272546999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:43.305227558 +0000 UTC m=+6.768501979" watchObservedRunningTime="2026-11-03 16:13:43.343674416 +0000 UTC m=+6.806948835" Nov 03 16:13:44.036630 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:44.036594 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:44.036804 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:44.036728 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:44.037184 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:44.037157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:44.037318 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:44.037264 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:44.277510 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:44.277060 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wmmj" event={"ID":"d5a6779b-623f-4618-bc63-f9efc8008fa4","Type":"ContainerStarted","Data":"cc9977cca732022b8827e45be8bcc4be74da00cdf7fa0811880c54f1850a6047"} Nov 03 16:13:44.316039 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:44.315989 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wmmj" podStartSLOduration=1.818441027 podStartE2EDuration="7.315974522s" totalImagesPullingTime="5.497533495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:44.3149951 +0000 UTC m=+7.778269520" watchObservedRunningTime="2026-11-03 16:13:44.315974522 +0000 UTC m=+7.779248964" Nov 03 16:13:45.039547 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:45.037356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:45.039547 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.037524 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:45.281856 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:45.281739 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" event={"ID":"fe8cc36e-da75-4685-88ae-5fa8db05b7a3","Type":"ContainerStarted","Data":"c369e8fb9862a398a2f5e211cf863b00f87d28457ef83be60766e3adc21156d7"} Nov 03 16:13:45.735019 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:45.734181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:45.735019 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:45.734254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:45.735019 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.734407 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:45.735019 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.734503 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:13:53.734484088 +0000 UTC m=+17.197758541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:45.735019 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.734914 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:45.735019 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.734971 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret podName:e0165af3-4e86-4907-8919-6ec892f905b7 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:53.734955246 +0000 UTC m=+17.198229663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret") pod "global-pull-secret-syncer-h6g2k" (UID: "e0165af3-4e86-4907-8919-6ec892f905b7") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:45.835395 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:45.834774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:45.835395 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.834943 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:45.835395 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.834966 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:45.835395 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.834980 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lc4lw for pod openshift-network-diagnostics/network-check-target-p48p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:45.835395 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:45.835048 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw podName:3f8aeec1-65f6-482a-b42d-873c5e7c1903 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:53.835030708 +0000 UTC m=+17.298305127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lc4lw" (UniqueName: "kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw") pod "network-check-target-p48p4" (UID: "3f8aeec1-65f6-482a-b42d-873c5e7c1903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:46.036773 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.036746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:46.036927 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.036752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:46.036927 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:46.036854 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:46.037013 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:46.036948 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:46.075319 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.075290 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:13:46.286877 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.286663 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" event={"ID":"fe8cc36e-da75-4685-88ae-5fa8db05b7a3","Type":"ContainerStarted","Data":"38154dd9486ba1113848aab2f3ae45642b4ca9ef3cbf740c8201991bcead8a95"} Nov 03 16:13:46.288836 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.288800 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w6prk" event={"ID":"174d66ce-30d8-4470-91e1-058c1e7e0c3b","Type":"ContainerStarted","Data":"494a8f7ab6c3438ebf185a0cc4df26e3e1c3ea6d1887f5431bede7e2891d1425"} Nov 03 16:13:46.302743 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.302711 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:46.303341 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.303318 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:13:46.368510 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:46.368449 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w6prk" podStartSLOduration=1.5868485190000001 podStartE2EDuration="9.368409252s" totalImagesPullingTime="7.781560733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:46.330846751 +0000 UTC m=+9.794121182" watchObservedRunningTime="2026-11-03 16:13:46.368409252 +0000 UTC m=+9.831683675" Nov 03 16:13:47.033484 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.033363 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:13:46.075313894Z","UUID":"96cb0c30-bfa8-4058-a8a6-72e3e97a6e97","Handler":null,"Name":"","Endpoint":""} Nov 03 16:13:47.035165 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.035138 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:13:47.035298 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.035172 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:13:47.042131 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.042107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:47.042290 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:47.042252 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:47.293839 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.293800 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="632eaabcdc03b29139d989d2c34ad92edf4e5f826f603d8a419e09f41d496370" exitCode=0 Nov 03 16:13:47.294286 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.293901 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerDied","Data":"632eaabcdc03b29139d989d2c34ad92edf4e5f826f603d8a419e09f41d496370"} Nov 03 16:13:47.297735 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.297134 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" event={"ID":"fe8cc36e-da75-4685-88ae-5fa8db05b7a3","Type":"ContainerStarted","Data":"5a51c10c6d1ab4023987e13933cb0a83e603e48df93fbf40bcaa91a96a876437"} Nov 03 16:13:47.300119 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.300097 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7dwpd" event={"ID":"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c","Type":"ContainerStarted","Data":"a7824263dfb5a7e972438c7491e914aa97794b13d79fe8bb4b19c20bed68547b"} Nov 03 16:13:47.378188 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:47.378131 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7dwpd" podStartSLOduration=1.883590512 podStartE2EDuration="10.378113326s" totalImagesPullingTime="8.494522814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:47.377593646 +0000 UTC m=+10.840868068" watchObservedRunningTime="2026-11-03 16:13:47.378113326 +0000 UTC m=+10.841387747" Nov 03 16:13:48.037041 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.036606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:48.037041 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.036626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:48.037041 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:48.036744 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:48.037041 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:48.036772 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:48.305755 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.305500 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="bd8e1025112560b363a8c68a861efd7f38aa77bfbb5c944a14ea2e2d308d64bc" exitCode=0 Nov 03 16:13:48.306142 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.305557 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerDied","Data":"bd8e1025112560b363a8c68a861efd7f38aa77bfbb5c944a14ea2e2d308d64bc"} Nov 03 16:13:48.308081 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.307483 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fvnv7" event={"ID":"25ddfba7-7752-4fad-be36-3f8d29b64a2b","Type":"ContainerStarted","Data":"400af88548cad81fb0a08c719e1ee75311d8a2ff5c45a1b63a41901bd3390f2a"} Nov 03 16:13:48.348722 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.348654 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" podStartSLOduration=2.940061911 podStartE2EDuration="11.348634551s" totalImagesPullingTime="8.40857264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:47.422979204 +0000 UTC m=+10.886253625" watchObservedRunningTime="2026-11-03 16:13:48.348634551 +0000 UTC m=+11.811908973" Nov 03 16:13:48.386584 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:48.386520 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fvnv7" podStartSLOduration=3.156471058 podStartE2EDuration="11.386489921s" totalImagesPullingTime="8.230018863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:48.385869495 +0000 UTC m=+11.849143916" watchObservedRunningTime="2026-11-03 16:13:48.386489921 +0000 UTC m=+11.849764342" Nov 03 16:13:49.036415 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:49.036386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:49.036567 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:49.036538 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:49.312730 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:49.312692 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="0c1a19cde6170235e5274ccd40e030d023c8ed9709702d655932f2933911f636" exitCode=0 Nov 03 16:13:49.313332 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:49.312746 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerDied","Data":"0c1a19cde6170235e5274ccd40e030d023c8ed9709702d655932f2933911f636"} Nov 03 16:13:49.314921 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:49.314899 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" event={"ID":"580cbd81-dec7-4b31-ab6c-e660173aecb4","Type":"ContainerStarted","Data":"845e5c4f202fdaadfbfd4f475c439457b2c201497cca04d63a59040c2ce28e9f"} Nov 03 16:13:49.397159 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:49.397112 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" podStartSLOduration=2.259995429 podStartE2EDuration="12.397097854s" totalImagesPullingTime="10.137102425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:49.396762773 +0000 UTC m=+12.860037194" watchObservedRunningTime="2026-11-03 16:13:49.397097854 +0000 UTC m=+12.860372274" Nov 03 16:13:50.036631 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:50.036596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:50.036824 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:50.036724 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:50.037073 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:50.037054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:50.037195 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:50.037139 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:51.037598 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:51.037560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:51.038289 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:51.037714 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:51.322713 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:51.322643 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="20670fce2019736a17bb5ffc03364531c9f636c8c35442cf0ca4f100b372a1c8" exitCode=0 Nov 03 16:13:51.322713 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:51.322684 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerDied","Data":"20670fce2019736a17bb5ffc03364531c9f636c8c35442cf0ca4f100b372a1c8"} Nov 03 16:13:52.036841 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:52.036807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:52.037010 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:52.036938 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:52.037280 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:52.037263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:52.037369 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:52.037353 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:52.328327 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:52.328255 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="9e885af41c77ade738ebd3fc7f21e960f20c996810b73a302f5be701f4636b64" exitCode=0 Nov 03 16:13:52.328327 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:52.328317 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerDied","Data":"9e885af41c77ade738ebd3fc7f21e960f20c996810b73a302f5be701f4636b64"} Nov 03 16:13:53.037442 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:53.037234 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:53.037609 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.037521 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:53.333849 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:53.333767 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" event={"ID":"8d31c32d-0e82-44a9-b9f1-e38073708fe6","Type":"ContainerStarted","Data":"a0836c1f2e8860867fab3b1e6677a37e2139bb9cdee9d79551da25e552ac4f66"} Nov 03 16:13:53.402257 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:53.402211 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6x8qs" podStartSLOduration=6.404248562 podStartE2EDuration="16.402198882s" totalImagesPullingTime="9.99795032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:53.401624921 +0000 UTC m=+16.864899353" watchObservedRunningTime="2026-11-03 16:13:53.402198882 +0000 UTC m=+16.865473302" Nov 03 16:13:53.795088 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:53.795064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:53.795175 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:53.795149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:53.795235 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.795216 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:53.795269 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.795256 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:53.795308 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.795280 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.79526524 +0000 UTC m=+33.258539639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:53.795308 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.795302 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret podName:e0165af3-4e86-4907-8919-6ec892f905b7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.795287386 +0000 UTC m=+33.258561807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret") pod "global-pull-secret-syncer-h6g2k" (UID: "e0165af3-4e86-4907-8919-6ec892f905b7") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:53.896250 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:53.896227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:53.896370 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.896347 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:53.896469 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.896379 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:53.896469 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.896389 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lc4lw for pod openshift-network-diagnostics/network-check-target-p48p4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:53.896469 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:53.896451 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw podName:3f8aeec1-65f6-482a-b42d-873c5e7c1903 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.896436966 +0000 UTC m=+33.359711378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lc4lw" (UniqueName: "kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw") pod "network-check-target-p48p4" (UID: "3f8aeec1-65f6-482a-b42d-873c5e7c1903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:54.036577 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.036542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:54.036722 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.036586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:54.036722 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:54.036663 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:54.036793 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:54.036778 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:54.338822 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.338799 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:13:54.339294 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339108 2575 generic.go:356] "Generic (PLEG): container finished" podID="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" containerID="42bbe53a1ac2b879ac11ecc2c415f66c3bc097c48666ac4d5bb544c60c7ffe54" exitCode=1 Nov 03 16:13:54.339294 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339172 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"70ce8b01616277b41a5e4b52dd80366531349933f718e9bc9aec4ad990042420"} Nov 03 16:13:54.339294 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339244 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"b1c1c99baa3bbec4a199379316f7b2811e4e2c0b91b6da5559c6eecefe47b1ff"} Nov 03 16:13:54.339294 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339290 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"14ebd9c490523890bfef3a00de0529b571a9314e268b02505e2bf66b63724f20"} Nov 03 16:13:54.339443 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339334 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"e3087c29c6e361604734bf7f6a2ab3d3df489a78e1f137cfca6330702fc3743a"} Nov 03 16:13:54.339443 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339381 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerDied","Data":"42bbe53a1ac2b879ac11ecc2c415f66c3bc097c48666ac4d5bb544c60c7ffe54"} Nov 03 16:13:54.339497 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:54.339452 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"6de5fef38200819e383b99366b510a7be3e04fc2d6552ce2ea1e1bf28a72b2d0"} Nov 03 16:13:55.037161 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:55.036995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:55.037313 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:55.037237 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:56.036352 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:56.036328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:56.036689 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:56.036438 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:56.036689 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:56.036456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:56.036689 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:56.036549 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:56.345913 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:56.345888 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:13:56.346294 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:56.346268 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"8fad4bb515d05d2cb27fc5ade7c7aee03683047f0807d526caf71338064ced6f"} Nov 03 16:13:57.037474 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:57.037445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:57.037913 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:57.037557 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:58.037345 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.037306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:13:58.037531 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.037362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:13:58.037531 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:58.037466 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:13:58.037846 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:58.037575 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:13:58.354651 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.354584 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:13:58.354977 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.354956 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"f30cfa52eb90a368edf91aa3d9523a1ade4c66d9d71e0a466344df1cc3a4959b"} Nov 03 16:13:58.355224 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.355205 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:58.355372 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.355356 2575 scope.go:122] "RemoveContainer" containerID="42bbe53a1ac2b879ac11ecc2c415f66c3bc097c48666ac4d5bb544c60c7ffe54" Nov 03 16:13:58.371800 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:58.371781 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:59.037073 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.037037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:13:59.037315 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:13:59.037147 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:13:59.362819 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.362753 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:13:59.363556 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.363153 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" event={"ID":"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b","Type":"ContainerStarted","Data":"2548723c77ad26c7378eed183e59b9d0e54c2302134c38911815d08ffdd58e6d"} Nov 03 16:13:59.363556 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.363293 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:13:59.363696 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.363599 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:59.378625 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.378601 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:13:59.459712 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:13:59.459652 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" podStartSLOduration=7.403634428 podStartE2EDuration="22.459635914s" totalImagesPullingTime="15.056001486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:59.459070244 +0000 UTC m=+22.922344664" watchObservedRunningTime="2026-11-03 16:13:59.459635914 +0000 UTC m=+22.922910335" Nov 03 16:14:00.037287 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.037061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:00.037412 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.037061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:00.037412 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:00.037398 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:14:00.037515 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:00.037463 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:14:00.113410 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.113262 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p48p4"] Nov 03 16:14:00.143676 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.143646 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wfft8"] Nov 03 16:14:00.144157 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.144132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:00.144324 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:00.144296 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:14:00.145307 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.145252 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h6g2k"] Nov 03 16:14:00.365771 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.365694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:00.365771 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.365725 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:00.366260 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:00.365813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:00.366260 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:00.365860 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:14:00.366260 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:00.365938 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:14:01.368324 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:01.368298 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:02.036803 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:02.036774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:02.036967 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:02.036862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:02.036967 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:02.036889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:02.037071 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:02.036866 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:14:02.037071 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:02.036965 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:14:02.037071 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:02.037037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:14:02.059597 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:02.059568 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:02.059751 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:02.059675 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:02.060153 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:02.060137 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:03.407266 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:03.407236 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:03.407668 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:03.407478 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:03.418565 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:03.418528 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" podUID="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 16:14:03.427313 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:03.427284 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" podUID="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 16:14:04.037101 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:04.037067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:04.037101 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:04.037097 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:04.037325 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:04.037112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:04.037325 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:04.037180 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:14:04.037459 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:04.037314 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:14:04.037459 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:04.037413 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:14:05.715455 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:05.715408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:05.961119 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:05.961089 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:06.037225 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:06.037197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:06.037225 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:06.037213 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:06.037417 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:06.037207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:06.037417 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:06.037289 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h6g2k" podUID="e0165af3-4e86-4907-8919-6ec892f905b7" Nov 03 16:14:06.037417 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:06.037380 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfft8" podUID="fdc671ec-9c42-41b8-983f-9385a62b14ba" Nov 03 16:14:06.037557 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:06.037448 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p48p4" podUID="3f8aeec1-65f6-482a-b42d-873c5e7c1903" Nov 03 16:14:06.900518 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:06.900489 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeReady" Nov 03 16:14:06.900988 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:06.900628 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 16:14:06.967176 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:06.967145 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7z595"] Nov 03 16:14:07.010539 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.010510 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lv59d"] Nov 03 16:14:07.010718 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.010650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.022405 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.022385 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7z595"] Nov 03 16:14:07.022505 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.022444 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lv59d"] Nov 03 16:14:07.022540 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.022503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.026520 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.026497 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:14:07.026520 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.026522 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:14:07.026775 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.026761 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:14:07.026851 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.026839 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nnkq7\"" Nov 03 16:14:07.038217 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.038194 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:14:07.038819 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.038803 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s5c6x\"" Nov 03 16:14:07.090135 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.090103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-config-volume\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.090274 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.090165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnsh\" (UniqueName: \"kubernetes.io/projected/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-kube-api-access-cnnsh\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.090274 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.090260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-cert\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.090349 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.090289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlppm\" (UniqueName: \"kubernetes.io/projected/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-kube-api-access-zlppm\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.090387 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.090367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.090387 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.090385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-tmp-dir\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.126797 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.126773 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:14:07.191464 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.191394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-config-volume\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.191464 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.191462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnsh\" (UniqueName: \"kubernetes.io/projected/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-kube-api-access-cnnsh\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.191639 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.191619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-cert\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.191683 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.191668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlppm\" (UniqueName: \"kubernetes.io/projected/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-kube-api-access-zlppm\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.191743 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.191730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.191775 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.191760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-tmp-dir\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.191904 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:07.191887 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:07.191976 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:07.191965 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.69194719 +0000 UTC m=+31.155221601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:07.192032 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.192016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-tmp-dir\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.192067 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.192034 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-config-volume\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.195246 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.195219 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-cert\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.218660 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.218640 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlppm\" (UniqueName: \"kubernetes.io/projected/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-kube-api-access-zlppm\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.222095 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.222075 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnsh\" (UniqueName: \"kubernetes.io/projected/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-kube-api-access-cnnsh\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.329017 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.328991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:07.394126 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.394101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:07.613206 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.613162 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:07.695863 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.695827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:07.696028 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:07.695976 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:07.696068 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:07.696056 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:08.696040076 +0000 UTC m=+32.159314477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:07.711396 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:07.711369 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7z595"] Nov 03 16:14:07.714987 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:14:07.714939 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4e8bf7_c9f1_4a7d_b385_fbf0e3d5af34.slice/crio-3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394 WatchSource:0}: Error finding container 3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394: Status 404 returned error can't find the container with id 3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394 Nov 03 16:14:08.037089 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.037062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:08.037528 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.037101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:08.037528 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.037071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:08.053540 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.053521 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:14:08.053540 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.053521 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:14:08.053689 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.053676 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:14:08.053864 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.053849 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:14:08.066659 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.066639 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:14:08.066659 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.066646 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:14:08.066659 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.066658 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:14:08.384875 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.384786 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7z595" event={"ID":"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34","Type":"ContainerStarted","Data":"3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394"} Nov 03 16:14:08.702016 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:08.701938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:08.702188 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:08.702113 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:08.702250 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:08.702193 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:10.702174623 +0000 UTC m=+34.165449040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:09.149039 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.149005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:09.303063 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.303023 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:09.811274 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.811229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:09.811274 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:09.811277 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:09.811565 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:09.811346 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:14:41.81132789 +0000 UTC m=+65.274602304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : secret "metrics-daemon-secret" not found Nov 03 16:14:09.811565 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.811362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:09.814104 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.814076 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:09.911728 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.911687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:09.914511 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.914489 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:09.918210 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:09.918184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:10.203890 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:10.203849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:10.260101 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:10.260072 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h6g2k"] Nov 03 16:14:10.264089 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:14:10.264054 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0165af3_4e86_4907_8919_6ec892f905b7.slice/crio-bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb WatchSource:0}: Error finding container bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb: Status 404 returned error can't find the container with id bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb Nov 03 16:14:10.390564 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:10.390492 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h6g2k" event={"ID":"e0165af3-4e86-4907-8919-6ec892f905b7","Type":"ContainerStarted","Data":"bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb"} Nov 03 16:14:10.537874 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:10.537841 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p48p4"] Nov 03 16:14:10.541562 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:14:10.541537 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8aeec1_65f6_482a_b42d_873c5e7c1903.slice/crio-29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00 WatchSource:0}: Error finding container 29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00: Status 404 returned error can't find the container with id 29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00 Nov 03 16:14:10.718216 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:10.718151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:10.718344 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:10.718274 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:10.718344 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:10.718325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:14.718312242 +0000 UTC m=+38.181586653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:11.395932 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.395531 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7z595" event={"ID":"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34","Type":"ContainerStarted","Data":"488d7883eca1c597fe0e727efe255c4f6598f0ef35b4647ae1c4751580732fda"} Nov 03 16:14:11.397521 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.397491 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p48p4" event={"ID":"3f8aeec1-65f6-482a-b42d-873c5e7c1903","Type":"ContainerStarted","Data":"29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00"} Nov 03 16:14:11.427523 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.427495 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4vn9v"] Nov 03 16:14:11.443903 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.443854 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7z595" podStartSLOduration=2.408437951 podStartE2EDuration="5.443841364s" totalImagesPullingTime="3.035403413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:11.443063035 +0000 UTC m=+34.906337456" watchObservedRunningTime="2026-11-03 16:14:11.443841364 +0000 UTC m=+34.907115783" Nov 03 16:14:11.447228 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.447205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.453355 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.453285 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4vn9v"] Nov 03 16:14:11.465754 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.465587 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:14:11.465754 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.465688 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:14:11.465754 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.465709 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:14:11.466000 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.465764 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5snks\"" Nov 03 16:14:11.466000 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.465770 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:14:11.503300 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.503273 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4lmtv-lk8ms"] Nov 03 16:14:11.519880 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.519852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.522902 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.522834 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4lmtv-lk8ms"] Nov 03 16:14:11.524414 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.524390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fed31089-612a-4f01-a506-17c494034dea-data-volume\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.524513 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.524441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.524513 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.524473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.525375 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.525356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fed31089-612a-4f01-a506-17c494034dea-crio-socket\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.525502 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.525395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmkx\" (UniqueName: \"kubernetes.io/projected/fed31089-612a-4f01-a506-17c494034dea-kube-api-access-qxmkx\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.554284 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.554225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:11.566249 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.566213 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 16:14:11.566458 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.566442 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j4959\"" Nov 03 16:14:11.588392 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.588356 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 16:14:11.626604 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fed31089-612a-4f01-a506-17c494034dea-data-volume\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.626758 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.626758 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.626758 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.626922 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:11.626762 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:11.626922 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.626922 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:11.626822 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:12.12680335 +0000 UTC m=+35.590077749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:11.626922 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:11.626856 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:11.626922 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fed31089-612a-4f01-a506-17c494034dea-crio-socket\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.626922 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:11.626915 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:12.126897919 +0000 UTC m=+35.590172337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : configmap "kube-rbac-proxy" not found Nov 03 16:14:11.627281 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626940 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fed31089-612a-4f01-a506-17c494034dea-data-volume\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.627281 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmkx\" (UniqueName: \"kubernetes.io/projected/fed31089-612a-4f01-a506-17c494034dea-kube-api-access-qxmkx\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.627281 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.626989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.627281 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.627030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.627281 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.627061 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fed31089-612a-4f01-a506-17c494034dea-crio-socket\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.653940 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.653879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmkx\" (UniqueName: \"kubernetes.io/projected/fed31089-612a-4f01-a506-17c494034dea-kube-api-access-qxmkx\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:11.707184 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.707140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:11.727911 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.727885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.727911 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.727918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.728146 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.728004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.728146 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.728034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.728989 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.728963 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.729101 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.729022 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.729322 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.729302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.752976 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.752949 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.850707 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.850672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:11.985286 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:11.985254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:12.130870 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.130833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:12.131064 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.130891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:12.131064 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:12.130965 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:12.131064 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:12.131007 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:12.131064 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:12.131056 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.131034011 +0000 UTC m=+36.594308413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : configmap "kube-rbac-proxy" not found Nov 03 16:14:12.131245 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:12.131071 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.131064949 +0000 UTC m=+36.594339346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:12.189747 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.189714 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4lmtv-lk8ms"] Nov 03 16:14:12.192386 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.192320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:12.199310 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:14:12.199285 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f084ab_4d2e_43ad_9c21_4606655f3c4a.slice/crio-53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57 WatchSource:0}: Error finding container 53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57: Status 404 returned error can't find the container with id 53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57 Nov 03 16:14:12.402471 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.402415 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" event={"ID":"55f084ab-4d2e-43ad-9c21-4606655f3c4a","Type":"ContainerStarted","Data":"53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57"} Nov 03 16:14:12.484400 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.484307 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:12.491098 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.491069 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p"] Nov 03 16:14:12.523556 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.523519 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p"] Nov 03 16:14:12.523714 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.523635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:12.539920 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.539889 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6xtgh\"" Nov 03 16:14:12.614002 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.613975 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 16:14:12.614390 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.614371 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:12.635733 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.635702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nj9q\" (UniqueName: \"kubernetes.io/projected/51cce97f-6d35-490e-9fa8-1c5d4717e52f-kube-api-access-8nj9q\") pod \"migrator-66df5d496b-vbp5p\" (UID: \"51cce97f-6d35-490e-9fa8-1c5d4717e52f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:12.642830 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.642800 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:12.736642 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.736558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nj9q\" (UniqueName: \"kubernetes.io/projected/51cce97f-6d35-490e-9fa8-1c5d4717e52f-kube-api-access-8nj9q\") pod \"migrator-66df5d496b-vbp5p\" (UID: \"51cce97f-6d35-490e-9fa8-1c5d4717e52f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:12.768232 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.768188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nj9q\" (UniqueName: \"kubernetes.io/projected/51cce97f-6d35-490e-9fa8-1c5d4717e52f-kube-api-access-8nj9q\") pod \"migrator-66df5d496b-vbp5p\" (UID: \"51cce97f-6d35-490e-9fa8-1c5d4717e52f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:12.835861 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.835789 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:12.847402 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.847370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:12.991247 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:12.991173 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:13.140564 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.139734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:13.140564 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.139787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:13.140564 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:13.139983 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:13.140564 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:13.140048 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:15.140029695 +0000 UTC m=+38.603304117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:13.140564 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:13.140472 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:13.140564 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:13.140528 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:15.140513108 +0000 UTC m=+38.603787534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : configmap "kube-rbac-proxy" not found Nov 03 16:14:13.172406 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.172375 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5792n"] Nov 03 16:14:13.192208 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:14:13.192167 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cce97f_6d35_490e_9fa8_1c5d4717e52f.slice/crio-da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc WatchSource:0}: Error finding container da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc: Status 404 returned error can't find the container with id da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc Nov 03 16:14:13.192453 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.192408 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p"] Nov 03 16:14:13.192567 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.192551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.194334 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.194308 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5792n"] Nov 03 16:14:13.209901 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.209866 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 16:14:13.210457 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.210219 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 16:14:13.210457 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.210259 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 16:14:13.210457 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.210288 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 16:14:13.210457 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.210311 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7v7v8\"" Nov 03 16:14:13.341260 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.341177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99xc\" (UniqueName: \"kubernetes.io/projected/5d955129-bc07-4636-8ad6-b8bf0725b5a2-kube-api-access-q99xc\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.341260 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.341241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-key\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.341522 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.341358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-cabundle\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.407517 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.407481 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" event={"ID":"51cce97f-6d35-490e-9fa8-1c5d4717e52f","Type":"ContainerStarted","Data":"da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc"} Nov 03 16:14:13.442325 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.442290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99xc\" (UniqueName: \"kubernetes.io/projected/5d955129-bc07-4636-8ad6-b8bf0725b5a2-kube-api-access-q99xc\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.442531 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.442342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-key\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.442531 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.442371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-cabundle\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.443103 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.443081 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-cabundle\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.452797 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.446457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-key\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.467399 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.467371 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99xc\" (UniqueName: \"kubernetes.io/projected/5d955129-bc07-4636-8ad6-b8bf0725b5a2-kube-api-access-q99xc\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.525539 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.525288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:13.603043 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.602949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:13.769359 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.769282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:13.854630 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:13.854547 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5792n"] Nov 03 16:14:13.858853 ip-10-0-139-190 kubenswrapper[2575]: W1103 16:14:13.858816 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d955129_bc07_4636_8ad6_b8bf0725b5a2.slice/crio-7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327 WatchSource:0}: Error finding container 7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327: Status 404 returned error can't find the container with id 7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327 Nov 03 16:14:14.243066 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:14.242938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:14.411948 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:14.411911 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" event={"ID":"5d955129-bc07-4636-8ad6-b8bf0725b5a2","Type":"ContainerStarted","Data":"7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327"} Nov 03 16:14:14.658874 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:14.658846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:14.754772 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:14.754725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:14.754989 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:14.754963 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:14.755106 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:14.755042 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:22.755023364 +0000 UTC m=+46.218297784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:14.858788 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:14.858707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:15.158995 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.158909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:15.158995 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.158973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:15.159255 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:15.159038 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:15.159255 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:15.159108 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.159092542 +0000 UTC m=+42.622366944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : configmap "kube-rbac-proxy" not found Nov 03 16:14:15.159255 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:15.159128 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:15.159255 ip-10-0-139-190 kubenswrapper[2575]: E1103 16:14:15.159199 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.159183064 +0000 UTC m=+42.622457484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:15.417350 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.417267 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p48p4" event={"ID":"3f8aeec1-65f6-482a-b42d-873c5e7c1903","Type":"ContainerStarted","Data":"9d3298d0f6a8ac48284cbeb8cfbfbd697675b30f7576fb8c37158403c6d2e952"} Nov 03 16:14:15.417831 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.417373 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:15.419166 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.419139 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" event={"ID":"55f084ab-4d2e-43ad-9c21-4606655f3c4a","Type":"ContainerStarted","Data":"da12a960cec764eb36a5311d0cb81e4e9a9346270d21d58a154fdccec78b4d6f"} Nov 03 16:14:15.420860 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.420836 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" event={"ID":"51cce97f-6d35-490e-9fa8-1c5d4717e52f","Type":"ContainerStarted","Data":"942387f2a651c4426121951fcec7faf019392863e8e8da392983d9796d32e161"} Nov 03 16:14:15.460201 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.459607 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p48p4" podStartSLOduration=34.411615722 podStartE2EDuration="38.459587825s" totalImagesPullingTime="4.047972103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:15.458976167 +0000 UTC m=+38.922250588" watchObservedRunningTime="2026-11-03 16:14:15.459587825 +0000 UTC m=+38.922862245" Nov 03 16:14:15.501358 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.501299 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" podStartSLOduration=1.716656535 podStartE2EDuration="3.501279771s" totalImagesPullingTime="1.784623236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:15.500770235 +0000 UTC m=+38.964044667" watchObservedRunningTime="2026-11-03 16:14:15.501279771 +0000 UTC m=+38.964554194" Nov 03 16:14:15.544538 ip-10-0-139-190 kubenswrapper[2575]: I1103 16:14:15.544477 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" podStartSLOduration=1.757567372 podStartE2EDuration="4.544458506s" totalImagesPullingTime="2.786891134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:15.543656118 +0000 UTC m=+39.006930539" watchObservedRunningTime="2026-11-03 16:14:15.544458506 +0000 UTC m=+39.007732925" Nov 03 16:14:16.222025 ip-10-0-139-190 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 16:14:16.238886 ip-10-0-139-190 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 16:14:16.239092 ip-10-0-139-190 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 16:14:16.239144 ip-10-0-139-190 systemd[1]: kubelet.service: Consumed 2.429s CPU time. Nov 03 16:14:16.254096 ip-10-0-139-190 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:14:16.354953 ip-10-0-139-190 kubenswrapper[4539]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:14:16.354953 ip-10-0-139-190 kubenswrapper[4539]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:14:16.354953 ip-10-0-139-190 kubenswrapper[4539]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:14:16.354953 ip-10-0-139-190 kubenswrapper[4539]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:14:16.358976 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358865 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.358976 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358976 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.358976 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358980 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.358976 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358984 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358988 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358991 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358994 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.358998 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359003 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359015 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359019 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359026 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359029 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359032 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359035 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359038 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359041 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359044 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359046 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359049 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359053 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359057 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359062 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.359201 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359076 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359085 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359113 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359117 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359120 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359123 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359127 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359130 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359132 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359135 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359139 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359141 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359144 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359150 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359153 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359155 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359160 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359165 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359170 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.359871 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359176 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359181 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359185 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359189 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359191 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359194 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359199 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359202 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359205 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359208 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359211 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359214 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359217 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359220 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359222 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359225 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359234 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359237 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359240 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359245 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.360580 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359248 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359253 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359268 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359272 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359275 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359277 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359281 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359284 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359286 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359289 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359292 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359295 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359300 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359303 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359305 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359311 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359315 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359318 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359321 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359323 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.361081 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359326 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359329 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359333 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359343 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359347 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359352 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359355 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359357 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359361 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359370 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359373 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359376 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359379 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359382 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359384 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359387 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359392 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359395 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359397 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359400 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.361594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359402 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359405 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359407 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.359410 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361084 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361097 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361101 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361107 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361111 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361116 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361120 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361122 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361125 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361128 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361130 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361133 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361136 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361138 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361141 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.362091 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361143 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361146 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361148 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361151 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361154 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361157 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361159 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361162 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361164 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361167 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361169 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361172 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361174 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361177 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361183 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361190 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361195 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361198 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361202 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.362643 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361204 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361207 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361210 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361212 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361215 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361217 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361221 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361225 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361227 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361230 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361233 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361236 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361238 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361240 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361243 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361245 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361248 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361250 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361253 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361255 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.363209 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361259 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361263 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361268 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361272 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361276 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361282 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361285 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361288 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361290 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361293 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361296 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361298 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361301 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361303 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361306 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361308 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361311 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361314 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361317 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361320 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.363872 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361323 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361325 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361328 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361331 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361333 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361336 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361338 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361341 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361344 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361349 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361354 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361358 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361363 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361366 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361368 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361371 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361373 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361376 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361378 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361381 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.364514 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361384 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361387 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361390 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361393 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361395 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361398 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361400 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361403 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361405 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361407 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361410 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361412 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361985 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361993 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.361998 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362002 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362006 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362010 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362014 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.365223 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362017 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362020 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362023 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362025 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362028 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362031 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362034 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362037 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362039 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362042 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362044 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362046 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362050 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362052 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362055 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362057 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362059 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362062 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362064 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362067 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.365780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362069 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362072 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362074 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362077 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362079 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362082 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362084 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362087 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362090 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362092 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362095 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362098 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362101 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362104 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362106 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362108 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362159 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362164 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362166 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362169 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.366438 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362172 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362174 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362177 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362180 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362182 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362184 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362187 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362189 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362192 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362194 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362197 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362199 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362202 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362205 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362207 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362210 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362212 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362215 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362217 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.367255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362220 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362222 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362225 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362228 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362230 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362233 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362235 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362238 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362240 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362243 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362245 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362248 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362251 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362253 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362256 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362258 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362261 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362263 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362266 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362268 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.367832 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362271 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362274 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362278 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362283 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362287 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362291 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362295 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362299 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362303 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362307 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362312 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362317 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362321 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362324 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362326 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362330 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362333 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362337 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362339 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.368548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.362342 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362460 4539 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362473 4539 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362490 4539 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362497 4539 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362504 4539 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362510 4539 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362516 4539 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362522 4539 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362527 4539 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362532 4539 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362537 4539 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362542 4539 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362546 4539 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362549 4539 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362552 4539 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362555 4539 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362557 4539 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362560 4539 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362564 4539 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362567 4539 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362570 4539 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362573 4539 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362576 4539 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:14:16.369181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362580 4539 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362583 4539 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362586 4539 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362589 4539 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362593 4539 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362598 4539 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362603 4539 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362608 4539 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362612 4539 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362618 4539 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362623 4539 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362629 4539 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362633 4539 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362639 4539 flags.go:64] FLAG: --enable-server="true" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362642 4539 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362649 4539 flags.go:64] FLAG: --event-burst="100" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362655 4539 flags.go:64] FLAG: --event-qps="50" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362660 4539 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362665 4539 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362669 4539 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362677 4539 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362682 4539 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362688 4539 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362693 4539 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362697 4539 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:14:16.369940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362700 4539 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362703 4539 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362706 4539 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362709 4539 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362711 4539 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362714 4539 flags.go:64] FLAG: --feature-gates="" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362726 4539 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362731 4539 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362735 4539 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362740 4539 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362744 4539 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362749 4539 flags.go:64] FLAG: --help="false" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362753 4539 flags.go:64] FLAG: --hostname-override="ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362759 4539 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362763 4539 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362766 4539 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362770 4539 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362773 4539 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362779 4539 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362783 4539 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362789 4539 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362795 4539 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362800 4539 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362806 4539 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:14:16.370708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362811 4539 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362815 4539 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362820 4539 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362824 4539 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362829 4539 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362833 4539 flags.go:64] FLAG: --lock-file="" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362837 4539 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362841 4539 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362845 4539 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362855 4539 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362858 4539 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362862 4539 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362864 4539 flags.go:64] FLAG: --logging-format="text" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362867 4539 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362871 4539 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362873 4539 flags.go:64] FLAG: --manifest-url="" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362876 4539 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362881 4539 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362885 4539 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362889 4539 flags.go:64] FLAG: --max-pods="110" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362892 4539 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362895 4539 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362898 4539 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362901 4539 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362905 4539 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:14:16.371360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362908 4539 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362913 4539 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362920 4539 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362925 4539 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362928 4539 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362933 4539 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362937 4539 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362942 4539 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362946 4539 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362950 4539 flags.go:64] FLAG: --port="10250" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362955 4539 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362959 4539 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00356a433a01daf35" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362964 4539 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362968 4539 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362972 4539 flags.go:64] FLAG: --register-node="true" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362975 4539 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362981 4539 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362986 4539 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362990 4539 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.362995 4539 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363001 4539 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363012 4539 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363016 4539 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363018 4539 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363021 4539 flags.go:64] FLAG: --runonce="false" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363024 4539 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:14:16.372252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363027 4539 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363031 4539 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363034 4539 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363036 4539 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363040 4539 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363043 4539 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363047 4539 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363051 4539 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363055 4539 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363062 4539 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363066 4539 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363070 4539 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363075 4539 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363080 4539 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363088 4539 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363092 4539 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363098 4539 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363103 4539 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363106 4539 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363108 4539 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363111 4539 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363114 4539 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363117 4539 flags.go:64] FLAG: --v="2" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363127 4539 flags.go:64] FLAG: --version="false" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363131 4539 flags.go:64] FLAG: --vmodule="" Nov 03 16:14:16.373027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363135 4539 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363138 4539 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363303 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363309 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363314 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363318 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363322 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363326 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363330 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363334 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363338 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363342 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363348 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363352 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363357 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363359 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363363 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363366 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363368 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363371 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.373917 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363373 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363377 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363382 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363385 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363387 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363390 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363392 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363395 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363397 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363400 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363402 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363405 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363407 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363410 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363412 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363415 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363417 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363433 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363436 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.374546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363442 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363460 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363465 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363469 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363473 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363476 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363481 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363485 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363491 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363494 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363498 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363500 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363503 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363506 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363508 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363511 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363513 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363516 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363518 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363521 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.375149 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363524 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363526 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363528 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363531 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363533 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363537 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363539 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363542 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363544 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363546 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363549 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363551 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363554 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363556 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363558 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363561 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363563 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363566 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363568 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363572 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.375785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363576 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363582 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363586 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363590 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363594 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363598 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363602 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363606 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363609 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363613 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363618 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363621 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363623 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363626 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363628 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363631 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363634 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363636 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363639 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363641 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.376414 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363644 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363646 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363648 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363651 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363654 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363656 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363659 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363661 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363663 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.363691 4539 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363744 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363751 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363756 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363760 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363765 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363768 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.377256 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363770 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363773 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363775 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363779 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363783 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363788 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363793 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363796 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363800 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363805 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363810 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363814 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363819 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363823 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363827 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363830 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363832 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363835 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363838 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363840 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.377827 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363842 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363845 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363849 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363853 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363857 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363861 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363865 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363869 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363875 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363880 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363886 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363890 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363893 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363895 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363898 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363901 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363905 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363910 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363914 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363918 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.378548 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363924 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363929 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363934 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363937 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363939 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363942 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363944 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363948 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363950 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363952 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363955 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363957 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363960 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363962 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363965 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363967 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363970 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363972 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363974 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363977 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.379128 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363980 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363983 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363987 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363992 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.363996 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364000 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364004 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364007 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364012 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364015 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364019 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364023 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364027 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364031 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364034 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364038 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364042 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364047 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364051 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.379775 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364056 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364060 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364065 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364070 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364074 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364077 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364079 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364082 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364084 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364087 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364089 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364094 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364098 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364101 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364103 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364106 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364109 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364112 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364116 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.380406 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364120 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.364124 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.369815 4539 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.369833 4539 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.369859 4539 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.369867 4539 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369952 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369961 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369968 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369975 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369980 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369985 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369989 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369993 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.369998 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370002 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370006 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370010 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370014 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.381156 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370018 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370023 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370028 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370033 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370037 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370041 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370045 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370050 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370054 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370058 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370063 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370067 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370072 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370076 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370080 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370084 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370088 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370092 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370096 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370100 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.381785 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370105 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370109 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370114 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370118 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370122 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370126 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370130 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370135 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370139 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370143 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370148 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370152 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370156 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370161 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370165 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370169 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370173 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370177 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370181 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.382352 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370185 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370190 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370194 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370198 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370203 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370207 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370211 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370215 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370219 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370223 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370228 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370232 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370237 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370241 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370245 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370250 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370254 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370258 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370262 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370269 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.382971 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370275 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370280 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370285 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370290 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370294 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370298 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370302 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370306 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370311 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370315 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370319 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370324 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370328 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370332 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370336 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370340 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370344 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370349 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370354 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370358 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.383624 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370362 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370366 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370370 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370374 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370378 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370382 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370387 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370392 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370397 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370401 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370405 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370409 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370413 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370418 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.370473 4539 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:14:16.384109 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370734 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370743 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370748 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370752 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370756 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370760 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370764 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370768 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370772 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370776 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370780 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370784 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370789 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370792 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370797 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370801 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370806 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370810 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370814 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370818 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.384490 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370822 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370826 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370830 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370834 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370838 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370843 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370847 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370851 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370855 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370859 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370863 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370867 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370872 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370876 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370880 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370885 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370889 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370892 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370896 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.385042 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370901 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370904 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370908 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370912 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370924 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370928 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370932 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370936 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370940 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370951 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370955 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370959 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370964 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370968 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370972 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370976 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370980 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370984 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370989 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.385596 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370994 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.370998 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371002 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371012 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371017 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371021 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371025 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371029 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371033 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371036 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371041 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371045 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371049 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371053 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371057 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371061 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371065 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371069 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371073 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.386207 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371077 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371081 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371086 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371090 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371101 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371106 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371110 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371114 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371120 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371126 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371132 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371137 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371142 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371146 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371150 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371154 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371161 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371167 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371172 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.386791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371176 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371181 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371185 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371189 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371193 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371197 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371201 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371205 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371209 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371213 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.371250 4539 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371323 4539 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371329 4539 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371333 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371339 4539 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.387334 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371345 4539 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371351 4539 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371355 4539 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371367 4539 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371372 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371379 4539 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371383 4539 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371387 4539 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371391 4539 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371396 4539 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371400 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371404 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371408 4539 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371413 4539 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371416 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371442 4539 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371447 4539 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371451 4539 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371455 4539 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371459 4539 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371463 4539 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.387798 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371467 4539 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371471 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371474 4539 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371478 4539 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371483 4539 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371488 4539 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371492 4539 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371496 4539 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371500 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371504 4539 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371508 4539 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371512 4539 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371516 4539 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371520 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371523 4539 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371534 4539 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371540 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371544 4539 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371548 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.388389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371552 4539 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371556 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371562 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371566 4539 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371570 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371575 4539 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371578 4539 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371582 4539 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371587 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371590 4539 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371594 4539 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371598 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371602 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371606 4539 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371611 4539 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371616 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371620 4539 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371624 4539 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371628 4539 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.388939 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371633 4539 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371637 4539 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371641 4539 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371645 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371649 4539 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371653 4539 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371657 4539 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371661 4539 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371665 4539 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371669 4539 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371682 4539 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371687 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371692 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371696 4539 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371700 4539 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371705 4539 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371710 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371713 4539 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371717 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371721 4539 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.389567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371725 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371729 4539 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371734 4539 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371738 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371742 4539 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371747 4539 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371752 4539 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371756 4539 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371760 4539 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371764 4539 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371768 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371772 4539 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371777 4539 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371781 4539 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371785 4539 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371789 4539 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371793 4539 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371797 4539 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371801 4539 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.390171 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371808 4539 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371815 4539 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371819 4539 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:16.371824 4539 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.371909 4539 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.373505 4539 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.373580 4539 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.373851 4539 server.go:1014] "Starting client certificate rotation" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.373989 4539 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.374056 4539 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:08:36 +0000 UTC" deadline="2028-08-17 23:37:31.011652358 +0000 UTC" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.374098 4539 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15679h23m14.637556938s" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.374640 4539 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.376461 4539 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.378160 4539 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.379982 4539 log.go:25] "Validated CRI v1 image API" Nov 03 16:14:16.390720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.380896 4539 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:14:16.391166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.382691 4539 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a33cdbbe-5cde-46e8-a6a5-fa8a37d32586:/dev/nvme0n1p3 d409c9a5-33f4-40f1-8efd-218884bf3e1e:/dev/nvme0n1p4] Nov 03 16:14:16.391223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.382720 4539 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05291fcf449d86065117ef2e707aadb11d15b311a87194fec51667682a7e89eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05291fcf449d86065117ef2e707aadb11d15b311a87194fec51667682a7e89eb/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/volumes/kubernetes.io~projected/kube-api-access-zlppm:{mountpoint:/var/lib/kubelet/pods/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/volumes/kubernetes.io~projected/kube-api-access-zlppm major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/174d66ce-30d8-4470-91e1-058c1e7e0c3b/volumes/kubernetes.io~projected/kube-api-access-cj6xr:{mountpoint:/var/lib/kubelet/pods/174d66ce-30d8-4470-91e1-058c1e7e0c3b/volumes/kubernetes.io~projected/kube-api-access-cj6xr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25ddfba7-7752-4fad-be36-3f8d29b64a2b/volumes/kubernetes.io~projected/kube-api-access-vshsf:{mountpoint:/var/lib/kubelet/pods/25ddfba7-7752-4fad-be36-3f8d29b64a2b/volumes/kubernetes.io~projected/kube-api-access-vshsf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f8aeec1-65f6-482a-b42d-873c5e7c1903/volumes/kubernetes.io~projected/kube-api-access-lc4lw:{mountpoint:/var/lib/kubelet/pods/3f8aeec1-65f6-482a-b42d-873c5e7c1903/volumes/kubernetes.io~projected/kube-api-access-lc4lw major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960/volumes/kubernetes.io~projected/kube-api-access-cnnsh:{mountpoint:/var/lib/kubelet/pods/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960/volumes/kubernetes.io~projected/kube-api-access-cnnsh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51cce97f-6d35-490e-9fa8-1c5d4717e52f/volumes/kubernetes.io~projected/kube-api-access-8nj9q:{mountpoint:/var/lib/kubelet/pods/51cce97f-6d35-490e-9fa8-1c5d4717e52f/volumes/kubernetes.io~projected/kube-api-access-8nj9q major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f084ab-4d2e-43ad-9c21-4606655f3c4a/volumes/kubernetes.io~projected/kube-api-access-z2pdx:{mountpoint:/var/lib/kubelet/pods/55f084ab-4d2e-43ad-9c21-4606655f3c4a/volumes/kubernetes.io~projected/kube-api-access-z2pdx major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~projected/kube-api-access-88cs9:{mountpoint:/var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~projected/kube-api-access-88cs9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volumes/kubernetes.io~projected/kube-api-access-pw96t:{mountpoint:/var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volumes/kubernetes.io~projected/kube-api-access-pw96t major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d955129-bc07-4636-8ad6-b8bf0725b5a2/volumes/kubernetes.io~projected/kube-api-access-q99xc:{mountpoint:/var/lib/kubelet/pods/5d955129-bc07-4636-8ad6-b8bf0725b5a2/volumes/kubernetes.io~projected/kube-api-access-q99xc major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d955129-bc07-4636-8ad6-b8bf0725b5a2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5d955129-bc07-4636-8ad6-b8bf0725b5a2/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bf7557b-7e3c-49a9-a522-1e46ba83b79f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8bf7557b-7e3c-49a9-a522-1e46ba83b79f/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d31c32d-0e82-44a9-b9f1-e38073708fe6/volumes/kubernetes.io~projected/kube-api-access-5tx5h:{mountpoint:/var/lib/kubelet/pods/8d31c32d-0e82-44a9-b9f1-e38073708fe6/volumes/kubernetes.io~projected/kube-api-access-5tx5h major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c/volumes/kubernetes.io~projected/kube-api-access-5988q:{mountpoint:/var/lib/kubelet/pods/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c/volumes/kubernetes.io~projected/kube-api-access-5988q major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5a6779b-623f-4618-bc63-f9efc8008fa4/volumes/kubernetes.io~projected/kube-api-access-vbjhl:{mountpoint:/var/lib/kubelet/pods/d5a6779b-623f-4618-bc63-f9efc8008fa4/volumes/kubernetes.io~projected/kube-api-access-vbjhl major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0165af3-4e86-4907-8919-6ec892f905b7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e0165af3-4e86-4907-8919-6ec892f905b7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc671ec-9c42-41b8-983f-9385a62b14ba/volumes/kubernetes.io~projected/kube-api-access-9xwbx:{mountpoint:/var/lib/kubelet/pods/fdc671ec-9c42-41b8-983f-9385a62b14ba/volumes/kubernetes.io~projected/kube-api-access-9xwbx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe8cc36e-da75-4685-88ae-5fa8db05b7a3/volumes/kubernetes.io~projected/kube-api-access-kvjpt:{mountpoint:/var/lib/kubelet/pods/fe8cc36e-da75-4685-88ae-5fa8db05b7a3/volumes/kubernetes.io~projected/kube-api-access-kvjpt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fed31089-612a-4f01-a506-17c494034dea/volumes/kubernetes.io~projected/kube-api-access-qxmkx:{mountpoint:/var/lib/kubelet/pods/fed31089-612a-4f01-a506-17c494034dea/volumes/kubernetes.io~projected/kube-api-access-qxmkx major:0 minor:218 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/354c5fd451a5aefd91312e801b9c7d750dacefad3a14cca724773155ad1c9ca2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8f4a8cb101b067b8c251d2f816c34054edd56820cfcdb660d936c4a16ad5f3a0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ed405411246469de4ca38ac31ce483f5058a504acabff2186963e2854eae38ca/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/89ec95705c6deccfefb3bbb8d70af1c6b766691b2fd20bb2ec54214e7ac71d7c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c90a2a27bb01756d11b5d8d1a066cf99b643d1faab901471c4ca383b176ac1ed/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cca11d3098e7f206d15817092b597cf52b2fe42511f1385e750c0c97dc4397ff/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0ec99b04b0c54d9609a643fd7aa30f42b68131833c25fe5dc03a20f6d15d8857/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/da50b941806674da880f40afa840ade7eb030cf1b47a1a6ba1c8a652bb043653/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/78c56744cc8b7ae75ed5ceb9dd4f3954c8ef5d223f196c6203620d55d35a758f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f4b7bad0afecc38944a099101e55d6c0294bfacc1ca455a620402cd74e2b2768/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/84c0bc15c7eeda352ccb7cd5d9dc47f4e4c25dd35a3430c9af374bfe63471927/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ca7560b1400027954a8491497c19478cadf521e629bf70ec878f6a67428a3913/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/56fe3015d2ee89d339f68848457017bcf7a4df82b1f144f4ba32ca09602dc1bb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/225bd8b5411d5dcac1f461c9b2995608634cf751ebb9cf9c7efe255d13e267bb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dd19ccc50ccf2a168f239a2ad1963a02bc4355ebec013d9784ac16c557d002a5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c50c3852352f3c176b11f398798ff72650dc45db40486a06dbf9a505334f187b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3bf461c95fc5eaab53f7e3bfc59cf229537e3bab8fa0078b0e9c60a92ed4bbed/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2b96116212e99c61dc552853442c881cb2be083e740e187db5db3ea94405536e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/be896b7024c8b1dceed90cc195627ae548376f40647c5b0fac917d33b32d2064/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0754bca873bd01c6c9dcd3b2daaf9d888418dcd6b60e90ab3a7b506d9e85032c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6b3d78af7260e2c4b844b0c8d8375980f9afd53cf51d141582b3c8b875ef4d48/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/02a4842ffe3d08f2054977670fc6854dc9c1091fba9730fe50a1a3b4e372d27b/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/5fc3b7dad3b5d1a5cb1de2cd4330b966db9f0405c97c762df9b9c9161d23fdec/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/7aeeb30ca0f925f6742be6ec4e75d8587c96e8c355fa9c688e9199c48d0494f6/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/01a48761bcaf7d437332b350962236ce047aa4d0166c571b7e3ca6a48eea45f5/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/297e478523eb2bb558a86f4e6e5d268043d515ffbe2f040cd63759c93857de7a/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/c8966b374f9923379fc0efd69ef2f8c320d2b76db530b2ce4d6874b84e2d5a7c/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/5c729a4f9b2e1887d62a559a255cdc36b5d55fe1fb12af924d107a418af9ad72/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/605c20138cb13617d7e046a230ddc5cc21917ead3075f1eccb90c58e578b9943/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/9c0010f30e017d567d4900caea80452b3a3fbc403e29784d9466656a3c7f71fe/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/63836ea3b1e651dac5914a9699d7ba70e12ed0eadc6d009b46fe0f31a1b6b1cc/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/0e358dd573febb9acb521b2c9386133aa634283d7234ee4038c65a674b690e98/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9311f1b1b4d72699d78470d549b59e1d0f4a0231943bb9962dd3f48c01147f99/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f6612bbb205f1372c142cbc5a90d609d76dac2e9d60a107a449b2f8059662430/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/13dce14cff6f19c1e8a5247804f2a143c0410a0b7345fadf518d1073cc29eb92/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c957098d95df5fe561ad9358396974ced20ce5f613fc815ba3c328205470ea0f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9b643cdfdbec680728f651c32ab9846793e5002f79bb71da012c73f16d6a3812/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0b8bd56c27dc7cd962da8f9ee42f18d879cd263f61cba223a78c445504485b2f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ae942cac8cdb866964c77f0a2e39cc62027609cee1b0bc5609eba1895bc24878/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/433db0f2f71d89befe2322b7d9b5cb5456bb0a63a7f39035f4bcb27135aaa980/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f8173b484828ce7754df9b4b1be1b5b200c3a10298057ab0e54a7025b1023673/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/b9ad784216eeaabc73ec33b7c819277ca8e9338386fffc3b73e50c88a3222dde/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 03 16:14:16.392890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.392367 4539 manager.go:217] Machine: {Timestamp:2026-11-03 16:14:16.391531543 +0000 UTC m=+0.084802558 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100899 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23724e6c73b018319dbeb90c4ce9a8 SystemUUID:ec23724e-6c73-b018-319d-beb90c4ce9a8 BootID:6b66123d-8682-4e82-83e8-1b8b25b10719 Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d955129-bc07-4636-8ad6-b8bf0725b5a2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8d31c32d-0e82-44a9-b9f1-e38073708fe6/volumes/kubernetes.io~projected/kube-api-access-5tx5h DeviceMajor:0 DeviceMinor:62 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcfd6462fc1b1e2ef2f46a664721493e441583336381addeb4c6037db5b871fb/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/55f084ab-4d2e-43ad-9c21-4606655f3c4a/volumes/kubernetes.io~projected/kube-api-access-z2pdx DeviceMajor:0 DeviceMinor:223 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/d5a6779b-623f-4618-bc63-f9efc8008fa4/volumes/kubernetes.io~projected/kube-api-access-vbjhl DeviceMajor:0 DeviceMinor:60 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/fe8cc36e-da75-4685-88ae-5fa8db05b7a3/volumes/kubernetes.io~projected/kube-api-access-kvjpt DeviceMajor:0 DeviceMinor:67 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57fe9e14a361b4a95c3e0bf072147c48125e1b83075733172977475da4c13c3a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/volumes/kubernetes.io~projected/kube-api-access-zlppm DeviceMajor:0 DeviceMinor:197 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960/volumes/kubernetes.io~projected/kube-api-access-cnnsh DeviceMajor:0 DeviceMinor:201 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0165af3-4e86-4907-8919-6ec892f905b7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/fed31089-612a-4f01-a506-17c494034dea/volumes/kubernetes.io~projected/kube-api-access-qxmkx DeviceMajor:0 DeviceMinor:218 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a1ed2830c33eb3b62b0f51df2d07af10bc51cfe5030fa938dcd72846d757327/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f8aeec1-65f6-482a-b42d-873c5e7c1903/volumes/kubernetes.io~projected/kube-api-access-lc4lw DeviceMajor:0 DeviceMinor:207 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bf7557b-7e3c-49a9-a522-1e46ba83b79f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c/volumes/kubernetes.io~projected/kube-api-access-5988q DeviceMajor:0 DeviceMinor:66 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25ddfba7-7752-4fad-be36-3f8d29b64a2b/volumes/kubernetes.io~projected/kube-api-access-vshsf DeviceMajor:0 DeviceMinor:59 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05291fcf449d86065117ef2e707aadb11d15b311a87194fec51667682a7e89eb/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc671ec-9c42-41b8-983f-9385a62b14ba/volumes/kubernetes.io~projected/kube-api-access-9xwbx DeviceMajor:0 DeviceMinor:64 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3749a74887b4152311ba35242777bcc86be0bbf5a0cd2cada199611eb3c5e394/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29e99e12b4f295538609b6931fdce75432e7b4bfa102ab82a04f7705c0f5ac00/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volumes/kubernetes.io~projected/kube-api-access-pw96t DeviceMajor:0 DeviceMinor:65 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/027a9ea722451dbbbbf65afaa2e568f7687f6a8c947127e5b281b3eb787b4557/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6bcb8be7e0c0214dfe2450cb4284010efdfeccf0583d9b6bfd5c5ea1d0485d8/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/860a2a37301e5d8858512c72af0db7e24701147167b8a67998a6a29c43f51cda/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e81382dd6109901d2ce4916d147674712a4a6558d2af1f512896af9a4f2c27af/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/51cce97f-6d35-490e-9fa8-1c5d4717e52f/volumes/kubernetes.io~projected/kube-api-access-8nj9q DeviceMajor:0 DeviceMinor:228 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/580cbd81-dec7-4b31-ab6c-e660173aecb4/volumes/kubernetes.io~projected/kube-api-access-88cs9 DeviceMajor:0 DeviceMinor:61 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/174d66ce-30d8-4470-91e1-058c1e7e0c3b/volumes/kubernetes.io~projected/kube-api-access-cj6xr DeviceMajor:0 DeviceMinor:63 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4180b303cb4e22dfe770d3cee3b6636f5172582bb7f6522f63d07670a0436bec/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da4f5ad261c8d7283aba1f66d1a7296f040d2af9d0d960c6ee0b552af69bc6cc/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/5d955129-bc07-4636-8ad6-b8bf0725b5a2/volumes/kubernetes.io~projected/kube-api-access-q99xc DeviceMajor:0 DeviceMinor:234 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea276fcb571d1d3498b2ce7742abcfd538a62936ad24bddad95c1db4dbac0cb7/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:29e99e12b4f2955 MacAddress:f2:ae:f4:a2:35:45 Speed:10000 Mtu:8901} {Name:3749a74887b4152 MacAddress:1e:4f:c0:d3:7f:00 Speed:10000 Mtu:8901} {Name:53ff1863eb7d782 MacAddress:0e:3e:02:5c:48:da Speed:10000 Mtu:8901} {Name:7a1ed2830c33eb3 MacAddress:06:4b:cb:3d:dd:87 Speed:10000 Mtu:8901} {Name:bcfd6462fc1b1e2 MacAddress:06:c0:3a:8f:3e:6b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:ae:3c:93 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:df:ec:b6:3d:ba Speed:0 Mtu:8901} {Name:da4f5ad261c8d72 MacAddress:2a:7b:ac:8a:f0:1c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:ae:3c:93 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9e:5c:ff:54:81:e1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:73:b6:51:ee:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:14:16.392890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.392893 4539 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:14:16.393181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.392952 4539 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:14:16.393326 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393300 4539 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:14:16.393515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393327 4539 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:14:16.393613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393519 4539 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:14:16.393613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393533 4539 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:14:16.393613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393547 4539 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:14:16.393613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393574 4539 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:14:16.393780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393696 4539 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:14:16.393943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.393930 4539 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:14:16.394075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394063 4539 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:14:16.394131 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394105 4539 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:14:16.394131 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394124 4539 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:14:16.394216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394138 4539 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:14:16.394216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394152 4539 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:14:16.394216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394164 4539 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:14:16.394930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394915 4539 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:14:16.394930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.394930 4539 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:14:16.395938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.395913 4539 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:14:16.406000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.405982 4539 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:14:16.406000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.405999 4539 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:14:16.406146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406111 4539 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 16:14:16.406351 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406335 4539 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:14:16.406442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406376 4539 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:14:16.406536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406524 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:14:16.406594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406545 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:14:16.406594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406556 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:14:16.406594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406566 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:14:16.406594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406576 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:14:16.406594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406586 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:14:16.406594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406595 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406604 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406617 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406627 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406648 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406662 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406697 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:14:16.406917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.406707 4539 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:14:16.410745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.410728 4539 server.go:1263] "Started kubelet" Nov 03 16:14:16.410859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.410813 4539 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:14:16.412030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.411123 4539 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:14:16.412030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.411334 4539 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:14:16.412030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.411725 4539 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:14:16.412030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.411988 4539 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:14:16.411313 ip-10-0-139-190 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:14:16.418314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418298 4539 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418700 4539 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418760 4539 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418773 4539 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418846 4539 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418963 4539 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:38 +0000 UTC" deadline="2028-05-08 11:00:21.332602825 +0000 UTC" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.418984 4539 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13242h46m4.913621635s" Nov 03 16:14:16.419189 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:16.419044 4539 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found" Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.419994 4539 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420008 4539 factory.go:55] Registering systemd factory Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420018 4539 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420271 4539 factory.go:153] Registering CRI-O factory Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420285 4539 factory.go:223] Registration of the crio container factory successfully Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420303 4539 factory.go:103] Registering Raw factory Nov 03 16:14:16.420464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420342 4539 manager.go:1199] Started watching for new ooms in manager Nov 03 16:14:16.420823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.420802 4539 manager.go:319] Starting recovery of all containers Nov 03 16:14:16.421626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421521 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" volumeName="kubernetes.io/projected/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-kube-api-access-pw96t" seLinuxMountContext="" Nov 03 16:14:16.421626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421579 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" volumeName="kubernetes.io/secret/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 16:14:16.421626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421594 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d955129-bc07-4636-8ad6-b8bf0725b5a2" volumeName="kubernetes.io/configmap/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-cabundle" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421605 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf7557b-7e3c-49a9-a522-1e46ba83b79f" volumeName="kubernetes.io/configmap/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-konnectivity-ca" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421827 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c" volumeName="kubernetes.io/empty-dir/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-tmp-dir" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421839 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c" volumeName="kubernetes.io/projected/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-kube-api-access-5988q" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421852 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f084ab-4d2e-43ad-9c21-4606655f3c4a" volumeName="kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421863 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d955129-bc07-4636-8ad6-b8bf0725b5a2" volumeName="kubernetes.io/projected/5d955129-bc07-4636-8ad6-b8bf0725b5a2-kube-api-access-q99xc" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421876 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d31c32d-0e82-44a9-b9f1-e38073708fe6" volumeName="kubernetes.io/projected/8d31c32d-0e82-44a9-b9f1-e38073708fe6-kube-api-access-5tx5h" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421889 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe8cc36e-da75-4685-88ae-5fa8db05b7a3" volumeName="kubernetes.io/projected/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kube-api-access-kvjpt" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421901 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed31089-612a-4f01-a506-17c494034dea" volumeName="kubernetes.io/projected/fed31089-612a-4f01-a506-17c494034dea-kube-api-access-qxmkx" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421914 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174d66ce-30d8-4470-91e1-058c1e7e0c3b" volumeName="kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cni-binary-copy" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421927 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cce97f-6d35-490e-9fa8-1c5d4717e52f" volumeName="kubernetes.io/projected/51cce97f-6d35-490e-9fa8-1c5d4717e52f-kube-api-access-8nj9q" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421939 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d31c32d-0e82-44a9-b9f1-e38073708fe6" volumeName="kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421954 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d31c32d-0e82-44a9-b9f1-e38073708fe6" volumeName="kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-binary-copy" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.421988 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0165af3-4e86-4907-8919-6ec892f905b7" volumeName="kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422000 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc671ec-9c42-41b8-983f-9385a62b14ba" volumeName="kubernetes.io/projected/fdc671ec-9c42-41b8-983f-9385a62b14ba-kube-api-access-9xwbx" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422014 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174d66ce-30d8-4470-91e1-058c1e7e0c3b" volumeName="kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-daemon-config" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422027 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ddfba7-7752-4fad-be36-3f8d29b64a2b" volumeName="kubernetes.io/configmap/25ddfba7-7752-4fad-be36-3f8d29b64a2b-iptables-alerter-script" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422040 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960" volumeName="kubernetes.io/configmap/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-config-volume" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422054 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f084ab-4d2e-43ad-9c21-4606655f3c4a" volumeName="kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422069 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580cbd81-dec7-4b31-ab6c-e660173aecb4" volumeName="kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-tuned" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422082 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d31c32d-0e82-44a9-b9f1-e38073708fe6" volumeName="kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422095 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34" volumeName="kubernetes.io/projected/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-kube-api-access-zlppm" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422108 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34" volumeName="kubernetes.io/secret/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-cert" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422121 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" volumeName="kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-script-lib" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422134 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" volumeName="kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-env-overrides" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422147 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d955129-bc07-4636-8ad6-b8bf0725b5a2" volumeName="kubernetes.io/secret/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-key" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422165 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5a6779b-623f-4618-bc63-f9efc8008fa4" volumeName="kubernetes.io/configmap/d5a6779b-623f-4618-bc63-f9efc8008fa4-serviceca" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422177 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f8aeec1-65f6-482a-b42d-873c5e7c1903" volumeName="kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422190 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f084ab-4d2e-43ad-9c21-4606655f3c4a" volumeName="kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422220 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580cbd81-dec7-4b31-ab6c-e660173aecb4" volumeName="kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-tmp" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422232 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="580cbd81-dec7-4b31-ab6c-e660173aecb4" volumeName="kubernetes.io/projected/580cbd81-dec7-4b31-ab6c-e660173aecb4-kube-api-access-88cs9" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422244 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" volumeName="kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-config" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422256 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5a6779b-623f-4618-bc63-f9efc8008fa4" volumeName="kubernetes.io/projected/d5a6779b-623f-4618-bc63-f9efc8008fa4-kube-api-access-vbjhl" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422267 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960" volumeName="kubernetes.io/empty-dir/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-tmp-dir" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422280 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf7557b-7e3c-49a9-a522-1e46ba83b79f" volumeName="kubernetes.io/secret/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-agent-certs" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422294 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fed31089-612a-4f01-a506-17c494034dea" volumeName="kubernetes.io/empty-dir/fed31089-612a-4f01-a506-17c494034dea-data-volume" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422305 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174d66ce-30d8-4470-91e1-058c1e7e0c3b" volumeName="kubernetes.io/projected/174d66ce-30d8-4470-91e1-058c1e7e0c3b-kube-api-access-cj6xr" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422325 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25ddfba7-7752-4fad-be36-3f8d29b64a2b" volumeName="kubernetes.io/projected/25ddfba7-7752-4fad-be36-3f8d29b64a2b-kube-api-access-vshsf" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422339 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960" volumeName="kubernetes.io/projected/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-kube-api-access-cnnsh" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422349 4539 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f084ab-4d2e-43ad-9c21-4606655f3c4a" volumeName="kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path" seLinuxMountContext="" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422360 4539 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:14:16.423202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.422367 4539 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:14:16.430015 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.429997 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:14:16.432792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.431602 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:14:16.448606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.448581 4539 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 16:14:16.450214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.450190 4539 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.450603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.450574 4539 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.457693 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.457657 4539 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:14:16.460558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.460534 4539 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:14:16.460558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.460557 4539 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:14:16.460731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.460578 4539 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:14:16.460731 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:16.460620 4539 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 16:14:16.461351 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.461298 4539 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.474042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.473977 4539 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.488360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.488322 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="9e885af41c77ade738ebd3fc7f21e960f20c996810b73a302f5be701f4636b64" exitCode=0 Nov 03 16:14:16.488528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.488363 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="20670fce2019736a17bb5ffc03364531c9f636c8c35442cf0ca4f100b372a1c8" exitCode=0 Nov 03 16:14:16.488528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.488375 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="0c1a19cde6170235e5274ccd40e030d023c8ed9709702d655932f2933911f636" exitCode=0 Nov 03 16:14:16.488528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.488385 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="bd8e1025112560b363a8c68a861efd7f38aa77bfbb5c944a14ea2e2d308d64bc" exitCode=0 Nov 03 16:14:16.488528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.488394 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="632eaabcdc03b29139d989d2c34ad92edf4e5f826f603d8a419e09f41d496370" exitCode=0 Nov 03 16:14:16.488528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.488403 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d31c32d-0e82-44a9-b9f1-e38073708fe6" containerID="00481ae7c3c5f47c67dd68ed7b0c16264e94b8bd8bdb5c6a55aed3a9d63b2d35" exitCode=0 Nov 03 16:14:16.497582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.497556 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:14:16.498504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.498467 4539 generic.go:356] "Generic (PLEG): container finished" podID="5ad67a8a-e10b-4af1-99ee-d97585bd6b6b" containerID="42bbe53a1ac2b879ac11ecc2c415f66c3bc097c48666ac4d5bb544c60c7ffe54" exitCode=1 Nov 03 16:14:16.507529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.507509 4539 manager.go:324] Recovery completed Nov 03 16:14:16.510300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.510273 4539 generic.go:356] "Generic (PLEG): container finished" podID="c7e318b374c3998927ea8465c3974849" containerID="991272ca83580ae41accb891c986bad156d28d7e153d71eaaaab56b73138566b" exitCode=0 Nov 03 16:14:16.524309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524292 4539 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:14:16.524309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524305 4539 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:14:16.524527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524321 4539 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:14:16.524527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524509 4539 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 16:14:16.524614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524522 4539 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 16:14:16.524614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524546 4539 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 16:14:16.524614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524551 4539 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 16:14:16.524614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524556 4539 policy_none.go:50] "Start" Nov 03 16:14:16.524614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524565 4539 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:14:16.524614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524574 4539 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:14:16.524776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524734 4539 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 16:14:16.524776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524741 4539 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 16:14:16.524776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.524747 4539 policy_none.go:44] "Start" Nov 03 16:14:16.529925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.529337 4539 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:14:16.535911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.535895 4539 manager.go:341] "Starting Device Plugin manager" Nov 03 16:14:16.536020 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:16.535929 4539 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:14:16.536020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.535940 4539 server.go:87] "Starting device plugin registration server" Nov 03 16:14:16.536285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536270 4539 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:14:16.536370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536287 4539 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:14:16.536538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536501 4539 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:14:16.536642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536584 4539 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:14:16.536642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536611 4539 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:14:16.536642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536615 4539 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:14:16.536837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.536654 4539 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:14:16.536603622Z","UUID":"f5869734-b910-4357-ad60-3f93ea00eaf5","Handler":null,"Name":"","Endpoint":""} Nov 03 16:14:16.538103 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:16.538082 4539 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:14:16.538261 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.538201 4539 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:14:16.538261 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.538245 4539 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:14:16.560727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.560676 4539 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"] Nov 03 16:14:16.561331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561288 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" event={"ID":"9f77174fff7f21e40f2a31f54c08055a","Type":"ContainerStarted","Data":"3e178774db051d43e8ba27c39a8e4d11836cdcbba47693ba43c3c9207c21b159"} Nov 03 16:14:16.561456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561336 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" event={"ID":"9f77174fff7f21e40f2a31f54c08055a","Type":"ContainerStarted","Data":"d462c8c2f18809767de232fcbea554ceff7783e6da861168eccd75adeb0302fb"} Nov 03 16:14:16.561456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561446 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"c7e318b374c3998927ea8465c3974849","Type":"ContainerStarted","Data":"50c8820ee3f80869f09f65f33788bbbd343612923e31d15db72d013d5152b462"} Nov 03 16:14:16.561573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561470 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"c7e318b374c3998927ea8465c3974849","Type":"ContainerDied","Data":"991272ca83580ae41accb891c986bad156d28d7e153d71eaaaab56b73138566b"} Nov 03 16:14:16.561573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561492 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"c7e318b374c3998927ea8465c3974849","Type":"ContainerStarted","Data":"5274b4cfe20376c6bab560f0d9c0ece15b83220c061b4ce416007b990117fb45"} Nov 03 16:14:16.561573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561541 4539 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.561573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.561562 4539 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.584248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.584213 4539 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:16.584374 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:16.584269 4539 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.584858 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.584843 4539 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:16.584962 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:16.584872 4539 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.637127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.637094 4539 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:14:16.638674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.638647 4539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:14:16.638806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.638692 4539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:14:16.638806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.638705 4539 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:14:16.638806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.638754 4539 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.650008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.649946 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f77174fff7f21e40f2a31f54c08055a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"9f77174fff7f21e40f2a31f54c08055a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.650008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.649985 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.650186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.650014 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.671617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.671592 4539 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.671733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.671679 4539 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.750335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.750247 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.750335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.750288 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f77174fff7f21e40f2a31f54c08055a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"9f77174fff7f21e40f2a31f54c08055a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.750335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.750314 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.750576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.750362 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.750576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.750365 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7e318b374c3998927ea8465c3974849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"c7e318b374c3998927ea8465c3974849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:16.750576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:16.750365 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f77174fff7f21e40f2a31f54c08055a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"9f77174fff7f21e40f2a31f54c08055a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:17.395160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.395130 4539 apiserver.go:52] "Watching apiserver" Nov 03 16:14:17.440030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.440000 4539 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:14:17.440514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.440492 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-h6g2k","openshift-cluster-node-tuning-operator/tuned-f9qd4","kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal","openshift-insights/periodic-gathering-4lmtv-lk8ms","openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p","openshift-multus/multus-w6prk","openshift-dns/node-resolver-7dwpd","openshift-image-registry/node-ca-9wmmj","openshift-network-operator/iptables-alerter-fvnv7","kube-system/konnectivity-agent-bvp5p","openshift-ingress-canary/ingress-canary-7z595","openshift-insights/insights-runtime-extractor-4vn9v","openshift-service-ca/service-ca-5fffc97bdf-5792n","openshift-dns/dns-default-lv59d","openshift-multus/multus-additional-cni-plugins-6x8qs","openshift-multus/network-metrics-daemon-wfft8","openshift-network-diagnostics/network-check-target-p48p4","openshift-ovn-kubernetes/ovnkube-node-gl2kp"] Nov 03 16:14:17.440984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.440966 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:17.447922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.447899 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.448721 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.448690 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.455469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.455453 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.455778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.455757 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.455903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.455801 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.455903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.455826 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.455903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.455835 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.456625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.456612 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:14:17.456755 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.456741 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:14:17.456903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.456762 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:14:17.456903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.456809 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hxhwt\"" Nov 03 16:14:17.456903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.456867 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:14:17.465047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.465032 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:14:17.465324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.465312 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.465358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.465332 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.467680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.467666 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:14:17.467732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.467680 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.467956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.467945 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 16:14:17.467998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.467987 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 16:14:17.468049 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.468029 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.468280 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.468269 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:14:17.468315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.468303 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s5c6x\"" Nov 03 16:14:17.468689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.468675 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:14:17.468745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.468695 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 16:14:17.468745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.468730 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.469237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.469223 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7v7v8\"" Nov 03 16:14:17.469293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.469251 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5xwqx\"" Nov 03 16:14:17.470251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470239 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.470589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470575 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:14:17.470638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470599 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:14:17.470680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470646 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.470729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470649 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:14:17.470729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470719 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5snks\"" Nov 03 16:14:17.470846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470756 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j4959\"" Nov 03 16:14:17.470846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470794 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tn5d9\"" Nov 03 16:14:17.470938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.470931 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:14:17.471332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471276 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:14:17.471332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471289 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.471332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471278 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.471332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471320 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.471332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471329 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.471551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471354 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tf624\"" Nov 03 16:14:17.471551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.471359 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:14:17.472094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472075 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9tss\"" Nov 03 16:14:17.472220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472136 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6xtgh\"" Nov 03 16:14:17.472220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472159 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.472220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472205 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:14:17.472220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472212 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:14:17.472398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472312 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6q76v\"" Nov 03 16:14:17.472398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472316 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.472398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472324 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.472398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472355 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.472398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472380 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:14:17.472398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.472383 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.475623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.475610 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.477881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.477866 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.478597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.478573 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:14:17.478677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.478609 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:14:17.479371 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.479355 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2k4df\"" Nov 03 16:14:17.479405 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.479381 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:14:17.479594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.479579 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n246q\"" Nov 03 16:14:17.480998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.480974 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:14:17.481471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.481457 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.481535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.481503 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:14:17.481535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.481511 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:14:17.481626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.481599 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlmt\"" Nov 03 16:14:17.481669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.481641 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nnkq7\"" Nov 03 16:14:17.482355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.482336 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 16:14:17.521932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.521906 4539 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:17.522132 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.522115 4539 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:17.522306 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.522290 4539 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:14:17.544037 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.544016 4539 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:17.544161 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.544060 4539 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" Nov 03 16:14:17.544456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.544443 4539 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:17.544487 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.544480 4539 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" Nov 03 16:14:17.553570 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553552 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmkx\" (UniqueName: \"kubernetes.io/projected/fed31089-612a-4f01-a506-17c494034dea-kube-api-access-qxmkx\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.553635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553577 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-log-socket\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.553635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553595 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-etc-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.553635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553608 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-bin\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.553635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553623 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-bin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.553815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553668 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tx5h\" (UniqueName: \"kubernetes.io/projected/8d31c32d-0e82-44a9-b9f1-e38073708fe6-kube-api-access-5tx5h\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.553815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553685 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:17.553815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553785 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-netns\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.553815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553805 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.553955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553877 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-hosts-file\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.553955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553927 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-cabundle\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:17.554016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.553968 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-kubelet-config\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.554078 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554066 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshsf\" (UniqueName: \"kubernetes.io/projected/25ddfba7-7752-4fad-be36-3f8d29b64a2b-kube-api-access-vshsf\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.554106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554094 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5a6779b-623f-4618-bc63-f9efc8008fa4-serviceca\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.554134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554113 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnsh\" (UniqueName: \"kubernetes.io/projected/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-kube-api-access-cnnsh\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.554134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554127 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-systemd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.554206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554140 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-system-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.554206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554156 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cnibin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.554206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554156 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-cabundle\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:17.554206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554178 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cnibin\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.554391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554247 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:17.554391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554290 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-systemd\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.554514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554396 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-sys\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.554514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554437 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5a6779b-623f-4618-bc63-f9efc8008fa4-serviceca\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.554514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554449 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjhl\" (UniqueName: \"kubernetes.io/projected/d5a6779b-623f-4618-bc63-f9efc8008fa4-kube-api-access-vbjhl\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.554514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554475 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.554514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554501 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-conf-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554525 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-daemon-config\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554557 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554633 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-socket-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554651 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-registration-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554672 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fed31089-612a-4f01-a506-17c494034dea-crio-socket\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554694 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fed31089-612a-4f01-a506-17c494034dea-data-volume\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.554732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554730 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-var-lib-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.554985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554750 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-daemon-config\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.554985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554775 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fed31089-612a-4f01-a506-17c494034dea-data-volume\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.554985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554819 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nj9q\" (UniqueName: \"kubernetes.io/projected/51cce97f-6d35-490e-9fa8-1c5d4717e52f-kube-api-access-8nj9q\") pod \"migrator-66df5d496b-vbp5p\" (UID: \"51cce97f-6d35-490e-9fa8-1c5d4717e52f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:17.554985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554859 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.554985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554936 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-modprobe-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.554985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554955 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-device-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.554987 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjpt\" (UniqueName: \"kubernetes.io/projected/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kube-api-access-kvjpt\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555028 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-config\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555115 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-script-lib\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555194 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-tmp-dir\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555216 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555221 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-config\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555232 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-cert\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555260 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-var-lib-kubelet\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555286 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555322 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-tmp-dir\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555367 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-key\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555389 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovnkube-script-lib\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555407 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-cert\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555437 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-config-volume\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555462 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555477 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-kubelet\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555491 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-slash\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555504 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-node-log\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555544 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cni-binary-copy\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555573 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-multus\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555581 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5d955129-bc07-4636-8ad6-b8bf0725b5a2-signing-key\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555601 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6xr\" (UniqueName: \"kubernetes.io/projected/174d66ce-30d8-4470-91e1-058c1e7e0c3b-kube-api-access-cj6xr\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555621 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-config-volume\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555630 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ddfba7-7752-4fad-be36-3f8d29b64a2b-iptables-alerter-script\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555648 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ddfba7-7752-4fad-be36-3f8d29b64a2b-host-slash\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.555666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555664 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555684 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw96t\" (UniqueName: \"kubernetes.io/projected/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-kube-api-access-pw96t\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555719 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cni-binary-copy\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555842 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-hostroot\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555849 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25ddfba7-7752-4fad-be36-3f8d29b64a2b-iptables-alerter-script\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555867 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-multus-certs\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555887 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-kubernetes\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555918 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwbx\" (UniqueName: \"kubernetes.io/projected/fdc671ec-9c42-41b8-983f-9385a62b14ba-kube-api-access-9xwbx\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.555985 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556024 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-sys-fs\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556061 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovn-node-metrics-cert\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556078 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysconfig\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556092 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-tmp\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556118 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556204 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556207 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556237 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-dbus\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556262 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556267 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-ovn-node-metrics-cert\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556206 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-tmp\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556289 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-lib-modules\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556336 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556337 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-tuned\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556371 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-konnectivity-ca\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556405 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556448 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-kubelet\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.556483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556483 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-tuned\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556544 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlppm\" (UniqueName: \"kubernetes.io/projected/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-kube-api-access-zlppm\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556571 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-conf\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556579 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-konnectivity-ca\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556596 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556638 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-systemd-units\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556672 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-ovn\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556697 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-socket-dir-parent\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556722 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-etc-kubernetes\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556752 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556775 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556791 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556806 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-k8s-cni-cncf-io\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556825 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556838 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-netd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556867 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5988q\" (UniqueName: \"kubernetes.io/projected/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-kube-api-access-5988q\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556934 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556963 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cs9\" (UniqueName: \"kubernetes.io/projected/580cbd81-dec7-4b31-ab6c-e660173aecb4-kube-api-access-88cs9\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.556986 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-agent-certs\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557036 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557068 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0165af3-4e86-4907-8919-6ec892f905b7-original-pull-secret\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557084 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557157 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557161 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557256 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-env-overrides\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557270 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bf7557b-7e3c-49a9-a522-1e46ba83b79f-agent-certs\") pod \"konnectivity-agent-bvp5p\" (UID: \"8bf7557b-7e3c-49a9-a522-1e46ba83b79f\") " pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557295 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-netns\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557389 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99xc\" (UniqueName: \"kubernetes.io/projected/5d955129-bc07-4636-8ad6-b8bf0725b5a2-kube-api-access-q99xc\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557407 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-run\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557389 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-env-overrides\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557442 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-host\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557447 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557481 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5a6779b-623f-4618-bc63-f9efc8008fa4-host\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.557576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557508 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-tmp-dir\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.558391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.557605 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-tmp-dir\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.573116 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.573094 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tx5h\" (UniqueName: \"kubernetes.io/projected/8d31c32d-0e82-44a9-b9f1-e38073708fe6-kube-api-access-5tx5h\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.575121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.575106 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmkx\" (UniqueName: \"kubernetes.io/projected/fed31089-612a-4f01-a506-17c494034dea-kube-api-access-qxmkx\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.575764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.575728 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnsh\" (UniqueName: \"kubernetes.io/projected/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-kube-api-access-cnnsh\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.576013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.575998 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshsf\" (UniqueName: \"kubernetes.io/projected/25ddfba7-7752-4fad-be36-3f8d29b64a2b-kube-api-access-vshsf\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.577781 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.577751 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjhl\" (UniqueName: \"kubernetes.io/projected/d5a6779b-623f-4618-bc63-f9efc8008fa4-kube-api-access-vbjhl\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.577878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.577846 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlppm\" (UniqueName: \"kubernetes.io/projected/0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34-kube-api-access-zlppm\") pod \"ingress-canary-7z595\" (UID: \"0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34\") " pod="openshift-ingress-canary/ingress-canary-7z595" Nov 03 16:14:17.577969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.577950 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5988q\" (UniqueName: \"kubernetes.io/projected/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-kube-api-access-5988q\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.578665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.578650 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nj9q\" (UniqueName: \"kubernetes.io/projected/51cce97f-6d35-490e-9fa8-1c5d4717e52f-kube-api-access-8nj9q\") pod \"migrator-66df5d496b-vbp5p\" (UID: \"51cce97f-6d35-490e-9fa8-1c5d4717e52f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vbp5p" Nov 03 16:14:17.578763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.578750 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4lw\" (UniqueName: \"kubernetes.io/projected/3f8aeec1-65f6-482a-b42d-873c5e7c1903-kube-api-access-lc4lw\") pod \"network-check-target-p48p4\" (UID: \"3f8aeec1-65f6-482a-b42d-873c5e7c1903\") " pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:17.579281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.579263 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6xr\" (UniqueName: \"kubernetes.io/projected/174d66ce-30d8-4470-91e1-058c1e7e0c3b-kube-api-access-cj6xr\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.579751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.579726 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjpt\" (UniqueName: \"kubernetes.io/projected/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kube-api-access-kvjpt\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.580163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.580127 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:17.581766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.581747 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") pod \"periodic-gathering-4lmtv-lk8ms\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:14:17.581826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.581769 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwbx\" (UniqueName: \"kubernetes.io/projected/fdc671ec-9c42-41b8-983f-9385a62b14ba-kube-api-access-9xwbx\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:17.581949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.581935 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99xc\" (UniqueName: \"kubernetes.io/projected/5d955129-bc07-4636-8ad6-b8bf0725b5a2-kube-api-access-q99xc\") pod \"service-ca-5fffc97bdf-5792n\" (UID: \"5d955129-bc07-4636-8ad6-b8bf0725b5a2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" Nov 03 16:14:17.582644 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.582628 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw96t\" (UniqueName: \"kubernetes.io/projected/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-kube-api-access-pw96t\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.596487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.596473 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cs9\" (UniqueName: \"kubernetes.io/projected/580cbd81-dec7-4b31-ab6c-e660173aecb4-kube-api-access-88cs9\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.657762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657704 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-systemd-units\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657734 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-ovn\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657757 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-socket-dir-parent\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657776 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-etc-kubernetes\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657804 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657804 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-systemd-units\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657818 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657825 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-ovn\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657838 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-socket-dir-parent\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657832 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-k8s-cni-cncf-io\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657857 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-k8s-cni-cncf-io\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657864 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657884 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-etc-kubernetes\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657884 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-netd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657899 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657915 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-netd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.657928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657925 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-netns\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657946 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-run\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657961 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-host\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657965 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-netns\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657975 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5a6779b-623f-4618-bc63-f9efc8008fa4-host\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.657993 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-log-socket\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658001 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-host\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658006 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-etc-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658026 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5a6779b-623f-4618-bc63-f9efc8008fa4-host\") pod \"node-ca-9wmmj\" (UID: \"d5a6779b-623f-4618-bc63-f9efc8008fa4\") " pod="openshift-image-registry/node-ca-9wmmj" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658028 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-bin\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658039 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-log-socket\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658040 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-etc-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658050 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-bin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658002 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-run\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658068 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658047 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-cni-bin\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658082 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-netns\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658101 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-netns\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658106 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658133 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-bin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658138 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658152 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-hosts-file\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658169 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-kubelet-config\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658193 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-systemd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658211 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-system-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658209 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c-hosts-file\") pod \"node-resolver-7dwpd\" (UID: \"cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c\") " pod="openshift-dns/node-resolver-7dwpd" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658224 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cnibin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658230 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-kubelet-config\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658241 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cnibin\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.658194 4539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658258 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-system-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658262 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-systemd\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658271 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-cnibin\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658245 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-run-systemd\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658271 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-cnibin\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658288 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-sys\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658296 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-systemd\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.658325 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.15830396 +0000 UTC m=+1.851574983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : secret "metrics-daemon-secret" not found Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658327 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-sys\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658339 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658364 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-conf-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658388 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658389 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-socket-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658407 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-conf-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658418 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-registration-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.658396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658452 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-socket-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658462 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fed31089-612a-4f01-a506-17c494034dea-crio-socket\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658492 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-registration-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658524 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-var-lib-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658576 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fed31089-612a-4f01-a506-17c494034dea-crio-socket\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658574 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-var-lib-openvswitch\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658581 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-modprobe-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658617 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-device-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658635 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658658 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-var-lib-kubelet\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658688 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d31c32d-0e82-44a9-b9f1-e38073708fe6-system-cni-dir\") pod \"multus-additional-cni-plugins-6x8qs\" (UID: \"8d31c32d-0e82-44a9-b9f1-e38073708fe6\") " pod="openshift-multus/multus-additional-cni-plugins-6x8qs" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658693 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-modprobe-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658710 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-device-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658723 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-var-lib-kubelet\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658701 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658803 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658831 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-kubelet\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658849 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-slash\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658867 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-node-log\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658898 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-node-log\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658900 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-kubelet\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.658920 4539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658913 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad67a8a-e10b-4af1-99ee-d97585bd6b6b-host-slash\") pod \"ovnkube-node-gl2kp\" (UID: \"5ad67a8a-e10b-4af1-99ee-d97585bd6b6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658933 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-multus\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658961 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ddfba7-7752-4fad-be36-3f8d29b64a2b-host-slash\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.658980 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.158964303 +0000 UTC m=+1.852235304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658983 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25ddfba7-7752-4fad-be36-3f8d29b64a2b-host-slash\") pod \"iptables-alerter-fvnv7\" (UID: \"25ddfba7-7752-4fad-be36-3f8d29b64a2b\") " pod="openshift-network-operator/iptables-alerter-fvnv7" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.658993 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-cni-multus\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659005 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659034 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-hostroot\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659067 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-multus-certs\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659082 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-kubernetes\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659094 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-hostroot\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659098 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-multus-cni-dir\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659097 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659120 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-run-multus-certs\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659124 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-sys-fs\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659129 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-kubernetes\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659139 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysconfig\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659158 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659160 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-sys-fs\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659176 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-dbus\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659186 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysconfig\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659191 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659206 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-lib-modules\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659217 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0165af3-4e86-4907-8919-6ec892f905b7-dbus\") pod \"global-pull-secret-syncer-h6g2k\" (UID: \"e0165af3-4e86-4907-8919-6ec892f905b7\") " pod="kube-system/global-pull-secret-syncer-h6g2k" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659223 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659243 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-kubelet\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659253 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fed31089-612a-4f01-a506-17c494034dea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659262 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-conf\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659277 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659290 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-lib-modules\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659300 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-d\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.659309 4539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659312 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174d66ce-30d8-4470-91e1-058c1e7e0c3b-host-var-lib-kubelet\") pod \"multus-w6prk\" (UID: \"174d66ce-30d8-4470-91e1-058c1e7e0c3b\") " pod="openshift-multus/multus-w6prk" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659327 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe8cc36e-da75-4685-88ae-5fa8db05b7a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jv4qt\" (UID: \"fe8cc36e-da75-4685-88ae-5fa8db05b7a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jv4qt" Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:17.659355 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.159344684 +0000 UTC m=+1.852615684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:17.660046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.659375 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/580cbd81-dec7-4b31-ab6c-e660173aecb4-etc-sysctl-conf\") pod \"tuned-f9qd4\" (UID: \"580cbd81-dec7-4b31-ab6c-e660173aecb4\") " pod="openshift-cluster-node-tuning-operator/tuned-f9qd4" Nov 03 16:14:17.751324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.751307 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:14:17.757230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:17.757209 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:18.164675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:18.164648 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:18.164801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:18.164703 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:18.164801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:18.164750 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:18.164868 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:18.164822 4539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:18.164914 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:18.164868 4539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:18.164914 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:18.164867 4539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:18.164914 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:18.164893 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.164872862 +0000 UTC m=+2.858143861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:18.165023 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:18.164928 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.164913856 +0000 UTC m=+2.858184872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : secret "metrics-daemon-secret" not found Nov 03 16:14:18.165023 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:18.164941 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.164935735 +0000 UTC m=+2.858206734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:18.525387 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:18.525351 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5792n" event={"ID":"5d955129-bc07-4636-8ad6-b8bf0725b5a2","Type":"ContainerStarted","Data":"3eab8c154e77865f62a6fc6480e2acbd5bc0115484adf0f1601e1b277d299b23"} Nov 03 16:14:19.065906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:19.065878 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7dwpd_cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c/dns-node-resolver/0.log" Nov 03 16:14:19.174482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:19.174445 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:19.174501 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:19.174526 4539 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:19.174595 4539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:19.174605 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs podName:fdc671ec-9c42-41b8-983f-9385a62b14ba nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.174588988 +0000 UTC m=+4.867859991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs") pod "network-metrics-daemon-wfft8" (UID: "fdc671ec-9c42-41b8-983f-9385a62b14ba") : secret "metrics-daemon-secret" not found Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:19.174528 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:19.174638 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.174626243 +0000 UTC m=+4.867897264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:19.174668 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:19.174654 4539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:19.174899 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:19.174702 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.174688941 +0000 UTC m=+4.867959943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:19.411571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:19.411491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:19.686994 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:19.686914 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:20.203227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:20.203202 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:20.213974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:20.213951 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:20.269124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:20.269091 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wmmj_d5a6779b-623f-4618-bc63-f9efc8008fa4/node-ca/0.log" Nov 03 16:14:20.801500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:20.801467 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:20.866264 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:20.866236 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7z595_0f4e8bf7-c9f1-4a7d-b385-fbf0e3d5af34/serve-healthcheck-canary/0.log" Nov 03 16:14:20.971240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:20.971201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:21.189225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.189152 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:21.189225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.189196 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:21.189381 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:21.189340 4539 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:21.189411 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.189394 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:21.189473 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:21.189440 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls podName:49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.189411141 +0000 UTC m=+8.882682141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls") pod "dns-default-lv59d" (UID: "49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960") : secret "dns-default-metrics-tls" not found Nov 03 16:14:21.189530 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:21.189500 4539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:21.189559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.189539 4539 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:14:21.189587 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:21.189558 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.189543841 +0000 UTC m=+8.882814851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:21.191641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.191625 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc671ec-9c42-41b8-983f-9385a62b14ba-metrics-certs\") pod \"network-metrics-daemon-wfft8\" (UID: \"fdc671ec-9c42-41b8-983f-9385a62b14ba\") " pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:21.287723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.287691 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:21.341852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.341827 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfft8" Nov 03 16:14:21.434094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.434066 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:21.466604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.466538 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vbp5p_51cce97f-6d35-490e-9fa8-1c5d4717e52f/migrator/0.log" Nov 03 16:14:21.467384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.467370 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:21.620597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.620565 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:21.632645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:21.632620 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wfft8"] Nov 03 16:14:21.635938 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:21.635903 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc671ec_9c42_41b8_983f_9385a62b14ba.slice/crio-96921ba98b7435434fd72c8000f36e22c504cb8856108faddd1f48edd74ecca4 WatchSource:0}: Error finding container 96921ba98b7435434fd72c8000f36e22c504cb8856108faddd1f48edd74ecca4: Status 404 returned error can't find the container with id 96921ba98b7435434fd72c8000f36e22c504cb8856108faddd1f48edd74ecca4 Nov 03 16:14:22.187312 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:22.187252 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:22.540056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:22.540020 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfft8" event={"ID":"fdc671ec-9c42-41b8-983f-9385a62b14ba","Type":"ContainerStarted","Data":"8c5b90ec334a6eb1a08b222ed2c208e4540de58ed5455dbf22d62a62fb4cecf4"} Nov 03 16:14:22.540160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:22.540074 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfft8" event={"ID":"fdc671ec-9c42-41b8-983f-9385a62b14ba","Type":"ContainerStarted","Data":"96921ba98b7435434fd72c8000f36e22c504cb8856108faddd1f48edd74ecca4"} Nov 03 16:14:22.788337 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:22.788299 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:22.976023 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:22.975979 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:23.387898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:23.387853 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:23.545221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:23.545186 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfft8" event={"ID":"fdc671ec-9c42-41b8-983f-9385a62b14ba","Type":"ContainerStarted","Data":"3cab09fc78d0d0a41a0dd940f6a5ab48c5c83d03d025131da4df9a3e86f32f96"} Nov 03 16:14:23.579551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:23.579521 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:24.183360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:24.183325 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:24.776439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:24.776386 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:24.990620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:24.990579 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:25.223934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.223850 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:25.223934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.223894 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:25.224119 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:25.223986 4539 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:25.224119 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:25.224037 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls podName:fed31089-612a-4f01-a506-17c494034dea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:33.224024563 +0000 UTC m=+16.917295576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4vn9v" (UID: "fed31089-612a-4f01-a506-17c494034dea") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:25.226356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.226335 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960-metrics-tls\") pod \"dns-default-lv59d\" (UID: \"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960\") " pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:25.248847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.248816 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:25.381999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.381969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:25.528569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.528544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:25.545271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.545201 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lv59d"] Nov 03 16:14:25.548341 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:25.548311 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49aa209e_24d5_4e0a_9e4a_a2cf9d2f6960.slice/crio-93ee97dfe4dad37ce8e1232b48e2dc4b03a078d1a526d09faf1b35f869c737f1 WatchSource:0}: Error finding container 93ee97dfe4dad37ce8e1232b48e2dc4b03a078d1a526d09faf1b35f869c737f1: Status 404 returned error can't find the container with id 93ee97dfe4dad37ce8e1232b48e2dc4b03a078d1a526d09faf1b35f869c737f1 Nov 03 16:14:25.691207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:25.691174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:26.180578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.180536 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:26.554599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.554530 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lv59d" event={"ID":"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960","Type":"ContainerStarted","Data":"93ee97dfe4dad37ce8e1232b48e2dc4b03a078d1a526d09faf1b35f869c737f1"} Nov 03 16:14:26.582907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.582866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:26.632933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.632909 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:26.633127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.633109 4539 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:26.791237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.791178 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:26.986242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:26.986159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:27.143871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.143833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:27.148698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.148676 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gl2kp" Nov 03 16:14:27.289302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.289279 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:27.441047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.441019 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:27.559437 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.559394 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lv59d" event={"ID":"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960","Type":"ContainerStarted","Data":"7601bc06cbedc7a9aca6b5db0ca372b8e126e32ee1122e060798e2fd1da8486e"} Nov 03 16:14:27.559862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.559841 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:27.559949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.559892 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lv59d" event={"ID":"49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960","Type":"ContainerStarted","Data":"32819a1936416de6deccdbc4d330984600792085271fa83742e003def9705935"} Nov 03 16:14:27.590883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:27.590849 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:28.189930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:28.189901 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:28.548754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:28.548728 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bvp5p" Nov 03 16:14:28.787601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:28.787567 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:29.004254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:29.004166 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:29.404316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:29.404277 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:29.592996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:29.592952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:29.791515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:29.791489 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:30.387399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:30.387369 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:30.534284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:30.534252 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:30.684365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:30.684281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:30.839939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:30.839911 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:30.995650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:30.995565 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:31.738906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:31.738873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:31.894807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:31.894777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:33.279156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:33.279122 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:33.281531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:33.281512 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fed31089-612a-4f01-a506-17c494034dea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4vn9v\" (UID: \"fed31089-612a-4f01-a506-17c494034dea\") " pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:33.349234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:33.349193 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4vn9v" Nov 03 16:14:33.638981 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:33.638906 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4vn9v"] Nov 03 16:14:33.641909 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:33.641881 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed31089_612a_4f01_a506_17c494034dea.slice/crio-518828fabdc752464fee364566e80eabcda78e475dafb3aedb18c2dd93eb1da4 WatchSource:0}: Error finding container 518828fabdc752464fee364566e80eabcda78e475dafb3aedb18c2dd93eb1da4: Status 404 returned error can't find the container with id 518828fabdc752464fee364566e80eabcda78e475dafb3aedb18c2dd93eb1da4 Nov 03 16:14:34.582952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.582921 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vn9v" event={"ID":"fed31089-612a-4f01-a506-17c494034dea","Type":"ContainerStarted","Data":"4d8fb6404ef086dc6e29807b7d0badec9ecad7e6b5688371833e49d9ae0bf077"} Nov 03 16:14:34.583332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.582980 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vn9v" event={"ID":"fed31089-612a-4f01-a506-17c494034dea","Type":"ContainerStarted","Data":"f2d742d942bce0eed784cf42701014d82e3a438e41e105b121415b47c8102e72"} Nov 03 16:14:34.583332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.583002 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vn9v" event={"ID":"fed31089-612a-4f01-a506-17c494034dea","Type":"ContainerStarted","Data":"518828fabdc752464fee364566e80eabcda78e475dafb3aedb18c2dd93eb1da4"} Nov 03 16:14:34.660160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.660127 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:34.712062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.712028 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vmkds"] Nov 03 16:14:34.715107 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.715086 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.736148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.736118 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vmkds"] Nov 03 16:14:34.745604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.745569 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qffp7\"" Nov 03 16:14:34.745604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.745596 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 16:14:34.745766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.745629 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 16:14:34.791189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.791153 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpbd\" (UniqueName: \"kubernetes.io/projected/56a09c07-a809-456c-912a-2697b95e1c63-kube-api-access-kfpbd\") pod \"downloads-6dd789c8b7-vmkds\" (UID: \"56a09c07-a809-456c-912a-2697b95e1c63\") " pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.791340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.791224 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56a09c07-a809-456c-912a-2697b95e1c63-tmp\") pod \"downloads-6dd789c8b7-vmkds\" (UID: \"56a09c07-a809-456c-912a-2697b95e1c63\") " pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.799229 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.799202 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66d98664dc-l7jt8"] Nov 03 16:14:34.802490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.802473 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.841934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.841876 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 16:14:34.842057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.841953 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 16:14:34.842395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.842382 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 16:14:34.842395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.842384 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qj6r5\"" Nov 03 16:14:34.842539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.842409 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 16:14:34.843015 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.842997 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66d98664dc-l7jt8"] Nov 03 16:14:34.856768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.856739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:34.892022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.891991 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-image-registry-private-configuration\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892027 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-trusted-ca\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892057 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-ca-trust-extracted\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892178 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdbj\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-kube-api-access-zgdbj\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892241 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-installation-pull-secrets\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892285 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpbd\" (UniqueName: \"kubernetes.io/projected/56a09c07-a809-456c-912a-2697b95e1c63-kube-api-access-kfpbd\") pod \"downloads-6dd789c8b7-vmkds\" (UID: \"56a09c07-a809-456c-912a-2697b95e1c63\") " pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.892460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892401 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-registry-tls\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892451 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56a09c07-a809-456c-912a-2697b95e1c63-tmp\") pod \"downloads-6dd789c8b7-vmkds\" (UID: \"56a09c07-a809-456c-912a-2697b95e1c63\") " pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.892560 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892487 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-bound-sa-token\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892560 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892537 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-registry-certificates\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.892899 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.892884 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56a09c07-a809-456c-912a-2697b95e1c63-tmp\") pod \"downloads-6dd789c8b7-vmkds\" (UID: \"56a09c07-a809-456c-912a-2697b95e1c63\") " pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.917634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.917610 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpbd\" (UniqueName: \"kubernetes.io/projected/56a09c07-a809-456c-912a-2697b95e1c63-kube-api-access-kfpbd\") pod \"downloads-6dd789c8b7-vmkds\" (UID: \"56a09c07-a809-456c-912a-2697b95e1c63\") " pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:34.993090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993052 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-registry-tls\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993106 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-bound-sa-token\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993129 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-registry-certificates\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993155 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-image-registry-private-configuration\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993171 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-trusted-ca\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993190 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-ca-trust-extracted\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993226 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdbj\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-kube-api-access-zgdbj\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.993333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.993301 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-installation-pull-secrets\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.994441 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.994361 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-registry-certificates\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.994686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.994638 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-trusted-ca\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.994958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.994921 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-ca-trust-extracted\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.996346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.996315 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-installation-pull-secrets\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.996505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.996478 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-image-registry-private-configuration\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:34.996617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:34.996530 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-registry-tls\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:35.015476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.015448 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-bound-sa-token\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:35.016943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.016923 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdbj\" (UniqueName: \"kubernetes.io/projected/21f70bc9-9ef1-4a86-9dbb-b513dca3ab28-kube-api-access-zgdbj\") pod \"image-registry-66d98664dc-l7jt8\" (UID: \"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28\") " pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:35.028906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.028883 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:35.048570 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.048543 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:35.113294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.113268 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:35.259137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.259104 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:35.353828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.353371 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vmkds"] Nov 03 16:14:35.357046 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:35.357005 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a09c07_a809_456c_912a_2697b95e1c63.slice/crio-cc5429872cbfdece90fb06ebf7363dd0c0fa4e65e14d2050676569c02388349f WatchSource:0}: Error finding container cc5429872cbfdece90fb06ebf7363dd0c0fa4e65e14d2050676569c02388349f: Status 404 returned error can't find the container with id cc5429872cbfdece90fb06ebf7363dd0c0fa4e65e14d2050676569c02388349f Nov 03 16:14:35.420378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.420333 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66d98664dc-l7jt8"] Nov 03 16:14:35.423122 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:35.423092 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f70bc9_9ef1_4a86_9dbb_b513dca3ab28.slice/crio-adb4b814a1e6d8878a693991314f87cb7dba1c8b60d3eda99b8373d6e2d51c0d WatchSource:0}: Error finding container adb4b814a1e6d8878a693991314f87cb7dba1c8b60d3eda99b8373d6e2d51c0d: Status 404 returned error can't find the container with id adb4b814a1e6d8878a693991314f87cb7dba1c8b60d3eda99b8373d6e2d51c0d Nov 03 16:14:35.587958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.587920 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" event={"ID":"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28","Type":"ContainerStarted","Data":"6467b7a41e1e7d7393c8e54a08008071fbff02dd10f97240a0df90960469623c"} Nov 03 16:14:35.588367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.587990 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" event={"ID":"21f70bc9-9ef1-4a86-9dbb-b513dca3ab28","Type":"ContainerStarted","Data":"adb4b814a1e6d8878a693991314f87cb7dba1c8b60d3eda99b8373d6e2d51c0d"} Nov 03 16:14:35.588772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.588750 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:35.590087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.590062 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vmkds" event={"ID":"56a09c07-a809-456c-912a-2697b95e1c63","Type":"ContainerStarted","Data":"cc5429872cbfdece90fb06ebf7363dd0c0fa4e65e14d2050676569c02388349f"} Nov 03 16:14:35.635584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:35.635501 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" podStartSLOduration=1.6354829610000001 podStartE2EDuration="1.635482961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:35.634654741 +0000 UTC m=+19.327925773" watchObservedRunningTime="2026-11-03 16:14:35.635482961 +0000 UTC m=+19.328753988" Nov 03 16:14:36.180377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.180334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:36.348203 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.348126 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:36.578220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.578191 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk"] Nov 03 16:14:36.586760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.586732 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:36.600683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.600604 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4vn9v" event={"ID":"fed31089-612a-4f01-a506-17c494034dea","Type":"ContainerStarted","Data":"c683ad0a91ab62aa16dad419d60da7038e5d653761e6f62735bdce502c350a59"} Nov 03 16:14:36.604129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.604108 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 16:14:36.604129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.604109 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jtl49\"" Nov 03 16:14:36.604617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.604578 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk"] Nov 03 16:14:36.611338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.611313 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:36.707329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.707300 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dbe0478a-bf9d-47a2-b337-4bea66f9d737-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-twdlk\" (UID: \"dbe0478a-bf9d-47a2-b337-4bea66f9d737\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:36.761228 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.761198 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:36.808469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.808411 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dbe0478a-bf9d-47a2-b337-4bea66f9d737-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-twdlk\" (UID: \"dbe0478a-bf9d-47a2-b337-4bea66f9d737\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:36.811439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.811387 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dbe0478a-bf9d-47a2-b337-4bea66f9d737-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-twdlk\" (UID: \"dbe0478a-bf9d-47a2-b337-4bea66f9d737\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:36.904684 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:36.904606 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:37.203945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:37.203864 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk"] Nov 03 16:14:37.565135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:37.565105 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lv59d" Nov 03 16:14:37.605989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:37.605916 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" event={"ID":"dbe0478a-bf9d-47a2-b337-4bea66f9d737","Type":"ContainerStarted","Data":"edec0e6b78336d00ad2f0c32de95e9d244f350a448194c1f553ea132789f17fb"} Nov 03 16:14:38.610614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:38.610582 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" event={"ID":"dbe0478a-bf9d-47a2-b337-4bea66f9d737","Type":"ContainerStarted","Data":"767ced4dfbd4c86e317a1fbcb80c0603aa92273fde9305e76b239eb3bfba5d71"} Nov 03 16:14:38.611081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:38.610890 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:38.616290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:38.616266 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" Nov 03 16:14:38.648465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:38.648401 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twdlk" podStartSLOduration=1.59450804 podStartE2EDuration="2.648389731s" totalImagesPullingTime="1.053881691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:38.648085643 +0000 UTC m=+22.341356682" watchObservedRunningTime="2026-11-03 16:14:38.648389731 +0000 UTC m=+22.341660752" Nov 03 16:14:39.706228 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.706195 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:39.795020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.794986 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2"] Nov 03 16:14:39.800958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.800928 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:39.815936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.815911 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:14:39.826234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.826140 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:14:39.826234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.826188 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 16:14:39.826475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.826247 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:14:39.826549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.826516 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z87vz\"" Nov 03 16:14:39.826604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.826581 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 16:14:39.837595 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.837571 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2"] Nov 03 16:14:39.858671 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.858640 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:39.932642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.932609 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:39.932826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.932653 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:39.932826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.932683 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:39.932942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:39.932844 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qptjv\" (UniqueName: \"kubernetes.io/projected/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-kube-api-access-qptjv\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.009850 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.009755 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:40.033884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.033853 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qptjv\" (UniqueName: \"kubernetes.io/projected/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-kube-api-access-qptjv\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.034074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.033903 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.034074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.034037 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.034190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.034084 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.034685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.034654 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.036870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.036845 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.036972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.036955 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.055736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.055711 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qptjv\" (UniqueName: \"kubernetes.io/projected/72f619ef-f2cb-4d6b-a106-d8abdf2c766c-kube-api-access-qptjv\") pod \"prometheus-operator-9cd89b74c-rx9n2\" (UID: \"72f619ef-f2cb-4d6b-a106-d8abdf2c766c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.116304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.116266 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" Nov 03 16:14:40.439131 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.439096 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2"] Nov 03 16:14:40.443251 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:40.443213 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f619ef_f2cb_4d6b_a106_d8abdf2c766c.slice/crio-fa7e00b22ca8d3b41c12fde3f462875954f70c8f63d406f7db39b82f9f8bc2ff WatchSource:0}: Error finding container fa7e00b22ca8d3b41c12fde3f462875954f70c8f63d406f7db39b82f9f8bc2ff: Status 404 returned error can't find the container with id fa7e00b22ca8d3b41c12fde3f462875954f70c8f63d406f7db39b82f9f8bc2ff Nov 03 16:14:40.620135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:40.620100 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" event={"ID":"72f619ef-f2cb-4d6b-a106-d8abdf2c766c","Type":"ContainerStarted","Data":"fa7e00b22ca8d3b41c12fde3f462875954f70c8f63d406f7db39b82f9f8bc2ff"} Nov 03 16:14:42.022192 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:42.022092 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:42.171940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:42.171874 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:42.589826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:42.589789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:42.628897 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:42.628861 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" event={"ID":"72f619ef-f2cb-4d6b-a106-d8abdf2c766c","Type":"ContainerStarted","Data":"4ba20db65f186da389d350fd22e6986335ed495389fd0f6c845f2a956a38362c"} Nov 03 16:14:42.629074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:42.628927 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" event={"ID":"72f619ef-f2cb-4d6b-a106-d8abdf2c766c","Type":"ContainerStarted","Data":"ef96fc65c74c0e003798ea70a0466ea896eda3175bf75f43866a9c560b043c0c"} Nov 03 16:14:42.687941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:42.687880 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rx9n2" podStartSLOduration=2.409930403 podStartE2EDuration="3.687864188s" totalImagesPullingTime="1.277933785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:42.677082406 +0000 UTC m=+26.370353428" watchObservedRunningTime="2026-11-03 16:14:42.687864188 +0000 UTC m=+26.381135209" Nov 03 16:14:44.182400 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.182348 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:44.413811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.413758 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:44.513109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.513021 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rrj9b"] Nov 03 16:14:44.516746 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.516720 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.576173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.576138 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:14:44.576366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.576323 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:14:44.576366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.576355 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:14:44.576520 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.576390 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cpmbj\"" Nov 03 16:14:44.617059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.617024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:44.671110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671081 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-root\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671146 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671244 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-metrics-client-ca\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671280 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk48n\" (UniqueName: \"kubernetes.io/projected/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-kube-api-access-tk48n\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671315 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671353 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-sys\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671433 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-textfile\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671520 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-tls\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.671566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.671542 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-wtmp\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.771934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.771837 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-root\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.771934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.771921 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.771949 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-metrics-client-ca\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.771960 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-root\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.771971 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk48n\" (UniqueName: \"kubernetes.io/projected/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-kube-api-access-tk48n\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772017 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772045 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-sys\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772068 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-textfile\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772116 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-sys\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772309 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-tls\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772359 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-wtmp\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772523 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-wtmp\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.772675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772623 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-metrics-client-ca\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.773331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.772809 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-textfile\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.773331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.773226 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.774889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.774866 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.775231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.775213 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-node-exporter-tls\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.795022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.794990 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk48n\" (UniqueName: \"kubernetes.io/projected/4b0a4814-d478-4c0a-8d41-9fb5a015cc4f-kube-api-access-tk48n\") pod \"node-exporter-rrj9b\" (UID: \"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f\") " pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.799624 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.799520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:44.834379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:44.834330 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrj9b" Nov 03 16:14:44.847311 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:14:44.847274 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0a4814_d478_4c0a_8d41_9fb5a015cc4f.slice/crio-a2010ba2c1d1985e81848973fd5c71da017d90a91f710ae8fa1e265849f8c290 WatchSource:0}: Error finding container a2010ba2c1d1985e81848973fd5c71da017d90a91f710ae8fa1e265849f8c290: Status 404 returned error can't find the container with id a2010ba2c1d1985e81848973fd5c71da017d90a91f710ae8fa1e265849f8c290 Nov 03 16:14:45.031989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:45.031950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:45.181921 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:45.181889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:45.602235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:45.602145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:45.641098 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:45.641050 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrj9b" event={"ID":"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f","Type":"ContainerStarted","Data":"a2010ba2c1d1985e81848973fd5c71da017d90a91f710ae8fa1e265849f8c290"} Nov 03 16:14:45.891248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:45.891161 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:46.041001 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:46.040966 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:46.652486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:46.652410 4539 generic.go:356] "Generic (PLEG): container finished" podID="4b0a4814-d478-4c0a-8d41-9fb5a015cc4f" containerID="9b6b6c491f43f28e64ef67692e62eb2a16027ab8aa025394e253a8e833139490" exitCode=0 Nov 03 16:14:46.653038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:46.652549 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrj9b" event={"ID":"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f","Type":"ContainerDied","Data":"9b6b6c491f43f28e64ef67692e62eb2a16027ab8aa025394e253a8e833139490"} Nov 03 16:14:46.903313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:46.903216 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:47.102299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.102257 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:47.278189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.278161 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:47.569080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.568992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:47.664748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.664706 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrj9b" event={"ID":"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f","Type":"ContainerStarted","Data":"ea74843d96e5b6d35ec2f71e1e68dae22b8f451b08eb69a93af16b25d1a11f76"} Nov 03 16:14:47.665211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.664820 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrj9b" event={"ID":"4b0a4814-d478-4c0a-8d41-9fb5a015cc4f","Type":"ContainerStarted","Data":"ff036d3a22f6dcdcbf6124bb25e190786e9094618041036eebeb7b12d983dc8b"} Nov 03 16:14:47.760158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.760113 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:47.949382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:47.949285 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:48.150790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:48.150754 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:49.142414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:49.142383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:49.292386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:49.292352 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:49.439585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:49.439486 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:51.469663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:51.469633 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p48p4" Nov 03 16:14:51.497752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:51.497723 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:51.508171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:51.508127 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rrj9b" podStartSLOduration=5.999215294 podStartE2EDuration="7.508110791s" totalImagesPullingTime="1.411290402s" totalInitContainerRuntime="97.605095ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:47.730358823 +0000 UTC m=+31.423629846" watchObservedRunningTime="2026-11-03 16:14:51.508110791 +0000 UTC m=+35.201381813" Nov 03 16:14:52.684240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:52.684201 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vmkds" event={"ID":"56a09c07-a809-456c-912a-2697b95e1c63","Type":"ContainerStarted","Data":"d3487e74ea937e57b367eb25c6eaa41e50228633f6f8203f26f7fc7d3cb0bfd2"} Nov 03 16:14:52.684827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:52.684806 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:52.695871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:52.695838 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vmkds" Nov 03 16:14:52.784283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:52.784221 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-vmkds" podStartSLOduration=2.288124158 podStartE2EDuration="18.784199899s" totalImagesPullingTime="16.496075741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:52.744487037 +0000 UTC m=+36.437758059" watchObservedRunningTime="2026-11-03 16:14:52.784199899 +0000 UTC m=+36.477470922" Nov 03 16:14:53.165988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:53.165954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:53.331233 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:53.331149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:56.170469 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:14:56.169859 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:14:56.831294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:56.831251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:57.096863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:57.096780 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:57.259997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:57.259960 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:57.611038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:57.611004 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66d98664dc-l7jt8" Nov 03 16:14:57.871375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:57.871271 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:58.021558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:58.021523 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:58.431854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:58.431817 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:58.603265 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:58.603168 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:14:59.937777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:14:59.937733 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:00.097546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:00.097513 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:05.343377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:05.343345 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:05.506771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:05.506739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:06.517588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:06.517547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:06.784515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:06.784487 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:12.024884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:12.024844 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:12.221933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:12.221904 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:13.879141 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:13.879055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:14.018063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:14.018031 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:22.089527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.089490 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:22.261598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.261570 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:22.607335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.607303 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:22.635099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.635066 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4lmtv-lk8ms"] Nov 03 16:15:22.635619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.635593 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" podUID="55f084ab-4d2e-43ad-9c21-4606655f3c4a" containerName="insights-gathering" containerID="cri-o://da12a960cec764eb36a5311d0cb81e4e9a9346270d21d58a154fdccec78b4d6f" gracePeriod=30 Nov 03 16:15:22.791398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.791366 4539 generic.go:356] "Generic (PLEG): container finished" podID="55f084ab-4d2e-43ad-9c21-4606655f3c4a" containerID="da12a960cec764eb36a5311d0cb81e4e9a9346270d21d58a154fdccec78b4d6f" exitCode=2 Nov 03 16:15:22.791567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.791437 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" event={"ID":"55f084ab-4d2e-43ad-9c21-4606655f3c4a","Type":"ContainerDied","Data":"da12a960cec764eb36a5311d0cb81e4e9a9346270d21d58a154fdccec78b4d6f"} Nov 03 16:15:22.801098 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.801065 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-w4zxh-fbdj2"] Nov 03 16:15:22.803174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.803140 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:22.816382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.816346 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.827368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.827340 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w4zxh-fbdj2"] Nov 03 16:15:22.861181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.861111 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd8j\" (UniqueName: \"kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.861181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.861142 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.861345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.861196 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.861345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.861265 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.922944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.922916 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:15:22.961783 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.961748 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") pod \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " Nov 03 16:15:22.961977 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.961802 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") pod \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " Nov 03 16:15:22.961977 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.961852 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") pod \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " Nov 03 16:15:22.961977 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.961886 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") pod \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\" (UID: \"55f084ab-4d2e-43ad-9c21-4606655f3c4a\") " Nov 03 16:15:22.962158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962064 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.962158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962097 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.962267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962176 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd8j\" (UniqueName: \"kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.962267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962204 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.962267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962238 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle" pod "55f084ab-4d2e-43ad-9c21-4606655f3c4a" (UID: "55f084ab-4d2e-43ad-9c21-4606655f3c4a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:22.962418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962293 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle" pod "55f084ab-4d2e-43ad-9c21-4606655f3c4a" (UID: "55f084ab-4d2e-43ad-9c21-4606655f3c4a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:22.962418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962324 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path" pod "55f084ab-4d2e-43ad-9c21-4606655f3c4a" (UID: "55f084ab-4d2e-43ad-9c21-4606655f3c4a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:22.962593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.962569 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.963235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.963194 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.963345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.963259 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:22.964698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.964672 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx" pod "55f084ab-4d2e-43ad-9c21-4606655f3c4a" (UID: "55f084ab-4d2e-43ad-9c21-4606655f3c4a"). InnerVolumeSpecName "kube-api-access-z2pdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:22.987252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:22.987221 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd8j\" (UniqueName: \"kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j\") pod \"periodic-gathering-w4zxh-fbdj2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:23.062984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.062945 4539 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/55f084ab-4d2e-43ad-9c21-4606655f3c4a-archives-path\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.062984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.062979 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pdx\" (UniqueName: \"kubernetes.io/projected/55f084ab-4d2e-43ad-9c21-4606655f3c4a-kube-api-access-z2pdx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.062984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.062989 4539 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-service-ca-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.063224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.062997 4539 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55f084ab-4d2e-43ad-9c21-4606655f3c4a-trusted-ca-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.143597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.143522 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:15:23.467359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.467284 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w4zxh-fbdj2"] Nov 03 16:15:23.470568 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:15:23.470546 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398da66c_071f_480f_84c0_cc4dedcef0b2.slice/crio-f739f9f96d63506ae545bf165ee5fe08925474dc75f5fb2b41930641ee905a27 WatchSource:0}: Error finding container f739f9f96d63506ae545bf165ee5fe08925474dc75f5fb2b41930641ee905a27: Status 404 returned error can't find the container with id f739f9f96d63506ae545bf165ee5fe08925474dc75f5fb2b41930641ee905a27 Nov 03 16:15:23.796771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.796739 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" Nov 03 16:15:23.796771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.796763 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4lmtv-lk8ms" event={"ID":"55f084ab-4d2e-43ad-9c21-4606655f3c4a","Type":"ContainerDied","Data":"53ff1863eb7d7824c10ac977f77efeb34de28085bd29dbb92c50a1491bce5f57"} Nov 03 16:15:23.797037 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.796827 4539 scope.go:122] "RemoveContainer" containerID="da12a960cec764eb36a5311d0cb81e4e9a9346270d21d58a154fdccec78b4d6f" Nov 03 16:15:23.798417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.798397 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" event={"ID":"398da66c-071f-480f-84c0-cc4dedcef0b2","Type":"ContainerStarted","Data":"993351c0c63a6d8ca3276d9b82c4ef6a42cb40e64bd9d937f24f05723e650c5c"} Nov 03 16:15:23.798417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.798458 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" event={"ID":"398da66c-071f-480f-84c0-cc4dedcef0b2","Type":"ContainerStarted","Data":"f739f9f96d63506ae545bf165ee5fe08925474dc75f5fb2b41930641ee905a27"} Nov 03 16:15:23.854227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.854194 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4lmtv-lk8ms"] Nov 03 16:15:23.864168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.864138 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4lmtv-lk8ms"] Nov 03 16:15:23.887758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.887726 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:23.893320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:23.893272 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" podStartSLOduration=1.893257528 podStartE2EDuration="1.893257528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:23.892570762 +0000 UTC m=+67.585841784" watchObservedRunningTime="2026-11-03 16:15:23.893257528 +0000 UTC m=+67.586528550" Nov 03 16:15:24.054711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:24.054638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:24.465801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:24.465731 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f084ab-4d2e-43ad-9c21-4606655f3c4a" path="/var/lib/kubelet/pods/55f084ab-4d2e-43ad-9c21-4606655f3c4a/volumes" Nov 03 16:15:26.155115 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:15:26.155074 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:15:27.030834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.030804 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:27.182343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.182313 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:27.356323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.355483 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5576bdf989-bqn8b"] Nov 03 16:15:27.356323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.356054 4539 cpu_manager.go:414] "containerMap: removing container" podUID="55f084ab-4d2e-43ad-9c21-4606655f3c4a" containerName="insights-gathering" Nov 03 16:15:27.356323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.356073 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f084ab-4d2e-43ad-9c21-4606655f3c4a" containerName="insights-gathering" Nov 03 16:15:27.356323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.356201 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f084ab-4d2e-43ad-9c21-4606655f3c4a" containerName="insights-gathering" Nov 03 16:15:27.373134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.373090 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.380860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.380834 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5576bdf989-bqn8b"] Nov 03 16:15:27.389908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.389877 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 16:15:27.389908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.389877 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-98qnk\"" Nov 03 16:15:27.390074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.389880 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 16:15:27.390074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.389955 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 16:15:27.390074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.389974 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 16:15:27.397491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397465 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-secret-telemeter-client\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397518 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397600 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397693 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-telemeter-client-tls\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397727 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6bg\" (UniqueName: \"kubernetes.io/projected/f93fc47d-fdb2-4f53-b45a-3023e5b82889-kube-api-access-qt6bg\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397752 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-federate-client-tls\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397783 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-metrics-client-ca\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.397894 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.397838 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-serving-certs-ca-bundle\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.431111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.431085 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 16:15:27.442333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.442303 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 16:15:27.498932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.498899 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.498943 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.498973 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-telemeter-client-tls\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.499020 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6bg\" (UniqueName: \"kubernetes.io/projected/f93fc47d-fdb2-4f53-b45a-3023e5b82889-kube-api-access-qt6bg\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.499053 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-federate-client-tls\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.499176 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-metrics-client-ca\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.499212 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-serving-certs-ca-bundle\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.499315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.499268 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-secret-telemeter-client\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.509954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.509922 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-metrics-client-ca\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.510103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.509979 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-serving-certs-ca-bundle\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.510346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.510321 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f93fc47d-fdb2-4f53-b45a-3023e5b82889-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.511727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.511703 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-federate-client-tls\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.511813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.511782 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.511858 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.511786 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-secret-telemeter-client\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.512224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.512203 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f93fc47d-fdb2-4f53-b45a-3023e5b82889-telemeter-client-tls\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.521207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.521184 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6bg\" (UniqueName: \"kubernetes.io/projected/f93fc47d-fdb2-4f53-b45a-3023e5b82889-kube-api-access-qt6bg\") pod \"telemeter-client-5576bdf989-bqn8b\" (UID: \"f93fc47d-fdb2-4f53-b45a-3023e5b82889\") " pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:27.721902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:27.721825 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" Nov 03 16:15:28.068364 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:28.068333 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5576bdf989-bqn8b"] Nov 03 16:15:28.072038 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:15:28.072013 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93fc47d_fdb2_4f53_b45a_3023e5b82889.slice/crio-4b0361d0263f6ec84a9b4dc225c0e49b96697c64049f0c4b45427343fba39126 WatchSource:0}: Error finding container 4b0361d0263f6ec84a9b4dc225c0e49b96697c64049f0c4b45427343fba39126: Status 404 returned error can't find the container with id 4b0361d0263f6ec84a9b4dc225c0e49b96697c64049f0c4b45427343fba39126 Nov 03 16:15:28.821103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:28.821028 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" event={"ID":"f93fc47d-fdb2-4f53-b45a-3023e5b82889","Type":"ContainerStarted","Data":"4b0361d0263f6ec84a9b4dc225c0e49b96697c64049f0c4b45427343fba39126"} Nov 03 16:15:30.829648 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:30.829610 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" event={"ID":"f93fc47d-fdb2-4f53-b45a-3023e5b82889","Type":"ContainerStarted","Data":"e544584f501eeb0085c6be3d0fc02c767a60317c6ad956dc05e044dff1726cee"} Nov 03 16:15:32.838362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:32.838323 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" event={"ID":"f93fc47d-fdb2-4f53-b45a-3023e5b82889","Type":"ContainerStarted","Data":"05dc413e9db67efef7be1577b227be88c7019a395a48790cc75fdd795a5aa18e"} Nov 03 16:15:32.838874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:32.838391 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" event={"ID":"f93fc47d-fdb2-4f53-b45a-3023e5b82889","Type":"ContainerStarted","Data":"8e7d9e20b9d82b32f5132a04b74935fd21879c21ad8cbbe4983d89f3ec8f5b4f"} Nov 03 16:15:32.904274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:32.904223 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5576bdf989-bqn8b" podStartSLOduration=2.26615192 podStartE2EDuration="5.904211482s" totalImagesPullingTime="3.638059562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:32.903947889 +0000 UTC m=+76.597218912" watchObservedRunningTime="2026-11-03 16:15:32.904211482 +0000 UTC m=+76.597482503" Nov 03 16:15:33.236277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:33.236194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:33.387473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:33.387410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:33.536625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:33.536595 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:56.154759 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:15:56.154719 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:15:58.066054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:58.066024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:15:59.501117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:15:59.501089 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:06.579487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:06.579449 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:09.171622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:09.171584 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:09.321468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:09.321405 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:09.806859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:09.806829 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:10.002270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:10.002240 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:10.150241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:10.150124 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:10.619897 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:10.619855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:10.770386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:10.770355 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:26.154776 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:16:26.154732 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:16:27.047840 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:27.047802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:27.212561 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:27.212530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:47.076167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:47.076128 4539 generic.go:356] "Generic (PLEG): container finished" podID="398da66c-071f-480f-84c0-cc4dedcef0b2" containerID="993351c0c63a6d8ca3276d9b82c4ef6a42cb40e64bd9d937f24f05723e650c5c" exitCode=0 Nov 03 16:16:47.076577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:47.076206 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" event={"ID":"398da66c-071f-480f-84c0-cc4dedcef0b2","Type":"ContainerDied","Data":"993351c0c63a6d8ca3276d9b82c4ef6a42cb40e64bd9d937f24f05723e650c5c"} Nov 03 16:16:48.206219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.206194 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:16:48.304139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304106 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j\" (UniqueName: \"kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j\") pod \"398da66c-071f-480f-84c0-cc4dedcef0b2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " Nov 03 16:16:48.304139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304142 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle\") pod \"398da66c-071f-480f-84c0-cc4dedcef0b2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " Nov 03 16:16:48.304383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304165 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path\") pod \"398da66c-071f-480f-84c0-cc4dedcef0b2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " Nov 03 16:16:48.304383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304195 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle\") pod \"398da66c-071f-480f-84c0-cc4dedcef0b2\" (UID: \"398da66c-071f-480f-84c0-cc4dedcef0b2\") " Nov 03 16:16:48.304594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304550 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path" pod "398da66c-071f-480f-84c0-cc4dedcef0b2" (UID: "398da66c-071f-480f-84c0-cc4dedcef0b2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:16:48.304633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304596 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle" pod "398da66c-071f-480f-84c0-cc4dedcef0b2" (UID: "398da66c-071f-480f-84c0-cc4dedcef0b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:48.304633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.304600 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle" pod "398da66c-071f-480f-84c0-cc4dedcef0b2" (UID: "398da66c-071f-480f-84c0-cc4dedcef0b2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:48.306318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.306298 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j" pod "398da66c-071f-480f-84c0-cc4dedcef0b2" (UID: "398da66c-071f-480f-84c0-cc4dedcef0b2"). InnerVolumeSpecName "kube-api-access-dmd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:16:48.404758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.404696 4539 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-service-ca-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:16:48.404758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.404720 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmd8j\" (UniqueName: \"kubernetes.io/projected/398da66c-071f-480f-84c0-cc4dedcef0b2-kube-api-access-dmd8j\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:16:48.404758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.404730 4539 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398da66c-071f-480f-84c0-cc4dedcef0b2-trusted-ca-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:16:48.404758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:48.404739 4539 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/398da66c-071f-480f-84c0-cc4dedcef0b2-archives-path\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:16:49.083392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:49.083359 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" event={"ID":"398da66c-071f-480f-84c0-cc4dedcef0b2","Type":"ContainerDied","Data":"f739f9f96d63506ae545bf165ee5fe08925474dc75f5fb2b41930641ee905a27"} Nov 03 16:16:49.083579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:49.083407 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f739f9f96d63506ae545bf165ee5fe08925474dc75f5fb2b41930641ee905a27" Nov 03 16:16:49.083579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:49.083368 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w4zxh-fbdj2" Nov 03 16:16:49.839179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:49.839139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:49.984410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:16:49.984371 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:16:56.155604 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:16:56.155566 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:17:06.761377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:17:06.761346 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:22.886216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:17:22.886181 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:23.037859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:17:23.037827 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:17:26.154796 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:17:26.154765 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:17:56.155413 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:17:56.155371 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:18:06.832651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:06.832559 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:16.474293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:16.474266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:26.154111 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:18:26.154073 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:18:28.804687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:28.804649 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:29.001383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:29.001352 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:30.341760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.341732 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj"] Nov 03 16:18:30.342237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.342117 4539 cpu_manager.go:414] "containerMap: removing container" podUID="398da66c-071f-480f-84c0-cc4dedcef0b2" containerName="insights-gathering" Nov 03 16:18:30.342237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.342136 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398da66c-071f-480f-84c0-cc4dedcef0b2" containerName="insights-gathering" Nov 03 16:18:30.342237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.342205 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="398da66c-071f-480f-84c0-cc4dedcef0b2" containerName="insights-gathering" Nov 03 16:18:30.345222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.345202 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.379603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.379557 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 16:18:30.379603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.379590 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 16:18:30.379801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.379668 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8n29r\"" Nov 03 16:18:30.412666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.412637 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj"] Nov 03 16:18:30.438887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.438858 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.438887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.438888 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2ct\" (UniqueName: \"kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.439067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.438916 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.442657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.442631 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt"] Nov 03 16:18:30.445969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.445951 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.469851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.469829 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt"] Nov 03 16:18:30.471629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.471604 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-flw2l\"" Nov 03 16:18:30.540288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540238 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.540484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540296 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7d5\" (UniqueName: \"kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.540484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540400 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.540484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540462 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2ct\" (UniqueName: \"kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.540622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540488 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.540622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540526 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.540774 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540757 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.540958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.540932 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.563160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.563129 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2ct\" (UniqueName: \"kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.641324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.641247 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.641324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.641314 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.641537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.641336 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7d5\" (UniqueName: \"kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.641682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.641664 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.641718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.641707 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.660293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.660252 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:18:30.664622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.664602 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7d5\" (UniqueName: \"kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.760917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.760890 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:18:30.948587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.948514 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:30.970342 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:30.970193 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj"] Nov 03 16:18:30.972043 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:30.972023 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468ffc7f_a01a_4257_9bd5_59b361ad3e5c.slice/crio-6fecb72ba141aad4b806f741bfe9857696a928c92165c9473a64b937851f7868 WatchSource:0}: Error finding container 6fecb72ba141aad4b806f741bfe9857696a928c92165c9473a64b937851f7868: Status 404 returned error can't find the container with id 6fecb72ba141aad4b806f741bfe9857696a928c92165c9473a64b937851f7868 Nov 03 16:18:31.063581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:31.063562 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt"] Nov 03 16:18:31.065469 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:31.065444 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d2adb5_b2e4_4c8e_90c1_bce49246cea9.slice/crio-16ef69e0520fc528f0a8c525ac9e2c41047b6f3fc4d141c72e3342aac5eb2c9c WatchSource:0}: Error finding container 16ef69e0520fc528f0a8c525ac9e2c41047b6f3fc4d141c72e3342aac5eb2c9c: Status 404 returned error can't find the container with id 16ef69e0520fc528f0a8c525ac9e2c41047b6f3fc4d141c72e3342aac5eb2c9c Nov 03 16:18:31.142785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:31.142763 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:31.412842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:31.412797 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" event={"ID":"13d2adb5-b2e4-4c8e-90c1-bce49246cea9","Type":"ContainerStarted","Data":"16ef69e0520fc528f0a8c525ac9e2c41047b6f3fc4d141c72e3342aac5eb2c9c"} Nov 03 16:18:31.414176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:31.414140 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" event={"ID":"468ffc7f-a01a-4257-9bd5-59b361ad3e5c","Type":"ContainerStarted","Data":"6fecb72ba141aad4b806f741bfe9857696a928c92165c9473a64b937851f7868"} Nov 03 16:18:32.362326 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.362288 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc"] Nov 03 16:18:32.366011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.365987 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.390151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.390114 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-zgk6p\"" Nov 03 16:18:32.394065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.394042 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc"] Nov 03 16:18:32.436136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.436090 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:32.456757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.456728 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.456901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.456768 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5rz\" (UniqueName: \"kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.456901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.456827 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.557849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.557814 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.558035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.557889 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.558035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.557932 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5rz\" (UniqueName: \"kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.558235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.558204 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.558496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.558478 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.581056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.581019 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5rz\" (UniqueName: \"kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:32.686442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:32.686336 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:18:33.005015 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:33.004825 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc"] Nov 03 16:18:33.008077 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:33.008043 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c58a485_0bca_423c_bf35_4853fb9fd674.slice/crio-047bec9ae438010ce95fbfcd7e7de04c05bf5b6c943a12e71c3ed4f0dd013957 WatchSource:0}: Error finding container 047bec9ae438010ce95fbfcd7e7de04c05bf5b6c943a12e71c3ed4f0dd013957: Status 404 returned error can't find the container with id 047bec9ae438010ce95fbfcd7e7de04c05bf5b6c943a12e71c3ed4f0dd013957 Nov 03 16:18:33.424573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:33.424533 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" event={"ID":"8c58a485-0bca-423c-bf35-4853fb9fd674","Type":"ContainerStarted","Data":"047bec9ae438010ce95fbfcd7e7de04c05bf5b6c943a12e71c3ed4f0dd013957"} Nov 03 16:18:34.382773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.382730 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc"] Nov 03 16:18:34.386895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.386783 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.401466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.401408 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-cq7hm\"" Nov 03 16:18:34.413618 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.413569 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc"] Nov 03 16:18:34.476001 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.475970 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.476187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.476021 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.476187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.476084 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rslmx\" (UniqueName: \"kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.479790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.479753 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz"] Nov 03 16:18:34.483764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.483745 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.498160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.497357 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qmfs2\"" Nov 03 16:18:34.507895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.507833 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz"] Nov 03 16:18:34.576937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.576853 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.576937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.576905 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbxq\" (UniqueName: \"kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.577160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.577004 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.577160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.577039 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.577160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.577063 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rslmx\" (UniqueName: \"kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.577160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.577100 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.577482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.577456 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.577646 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.577630 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.585320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.584898 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7"] Nov 03 16:18:34.588960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.588935 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.602472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.602401 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rslmx\" (UniqueName: \"kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.604193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.604161 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8tjtt\"" Nov 03 16:18:34.616226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.616176 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7"] Nov 03 16:18:34.678250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678175 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.678250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678217 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbxq\" (UniqueName: \"kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.678250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678242 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bkk\" (UniqueName: \"kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.678523 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678284 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.678523 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678312 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.678523 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678346 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.678726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.678705 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.679208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.679191 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.707887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.707849 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:18:34.713178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.713041 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbxq\" (UniqueName: \"kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.753998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.753960 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f"] Nov 03 16:18:34.758192 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.758170 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.773320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.773155 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pgpbs\"" Nov 03 16:18:34.780393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.780371 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.780528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.780464 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.780593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.780533 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bkk\" (UniqueName: \"kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.781084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.781062 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.781294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.781276 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.793737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.793686 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f"] Nov 03 16:18:34.802885 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.802861 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:18:34.808328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.808307 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bkk\" (UniqueName: \"kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.881054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.881023 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.881217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.881068 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpnx\" (UniqueName: \"kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.881217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.881161 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.910875 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.910842 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:18:34.983460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.982639 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.983460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.982697 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.983460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.982723 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpnx\" (UniqueName: \"kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.983667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.983580 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:34.983779 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:34.983740 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:35.008160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.008043 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpnx\" (UniqueName: \"kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:35.023074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.023042 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc"] Nov 03 16:18:35.030885 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:35.030441 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25e89b0_49eb_419a_9682_0f9b1143e0a2.slice/crio-15ab333b2ac6373ed1ab487775b5d293a3f51e3703e0a24f23cea85cd5a9c725 WatchSource:0}: Error finding container 15ab333b2ac6373ed1ab487775b5d293a3f51e3703e0a24f23cea85cd5a9c725: Status 404 returned error can't find the container with id 15ab333b2ac6373ed1ab487775b5d293a3f51e3703e0a24f23cea85cd5a9c725 Nov 03 16:18:35.076593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.076558 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:18:35.086940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.086912 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:35.118299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.118271 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz"] Nov 03 16:18:35.128274 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:35.128239 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61745b9c_ec5b_42c8_8a05_03c036cb48a0.slice/crio-bbf78c5269744af51e3235da6dc60b884a2729eedfca1ebe504702126f27b0c8 WatchSource:0}: Error finding container bbf78c5269744af51e3235da6dc60b884a2729eedfca1ebe504702126f27b0c8: Status 404 returned error can't find the container with id bbf78c5269744af51e3235da6dc60b884a2729eedfca1ebe504702126f27b0c8 Nov 03 16:18:35.241938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.241224 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7"] Nov 03 16:18:35.246304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.245979 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:35.247961 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:35.247908 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52e0c92_764f_49b9_9d87_879fa5c094ff.slice/crio-ce264582be253c8e4cc98c614fd3d0a1f7d19f47211c1ab03f7d4cf1d32f5e1a WatchSource:0}: Error finding container ce264582be253c8e4cc98c614fd3d0a1f7d19f47211c1ab03f7d4cf1d32f5e1a: Status 404 returned error can't find the container with id ce264582be253c8e4cc98c614fd3d0a1f7d19f47211c1ab03f7d4cf1d32f5e1a Nov 03 16:18:35.402791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.402761 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f"] Nov 03 16:18:35.407264 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:35.407205 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f5320d_e122_4c82_b47d_7e956dbe7579.slice/crio-dabef216a9b15ab3def09b26c637af6294b09ac22f869c81185c052ed45c153e WatchSource:0}: Error finding container dabef216a9b15ab3def09b26c637af6294b09ac22f869c81185c052ed45c153e: Status 404 returned error can't find the container with id dabef216a9b15ab3def09b26c637af6294b09ac22f869c81185c052ed45c153e Nov 03 16:18:35.447419 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.447286 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerStarted","Data":"bbf78c5269744af51e3235da6dc60b884a2729eedfca1ebe504702126f27b0c8"} Nov 03 16:18:35.451794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.451720 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" event={"ID":"f0f5320d-e122-4c82-b47d-7e956dbe7579","Type":"ContainerStarted","Data":"dabef216a9b15ab3def09b26c637af6294b09ac22f869c81185c052ed45c153e"} Nov 03 16:18:35.453659 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.453613 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" event={"ID":"e52e0c92-764f-49b9-9d87-879fa5c094ff","Type":"ContainerStarted","Data":"ce264582be253c8e4cc98c614fd3d0a1f7d19f47211c1ab03f7d4cf1d32f5e1a"} Nov 03 16:18:35.455440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.455379 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" event={"ID":"c25e89b0-49eb-419a-9682-0f9b1143e0a2","Type":"ContainerStarted","Data":"15ab333b2ac6373ed1ab487775b5d293a3f51e3703e0a24f23cea85cd5a9c725"} Nov 03 16:18:35.731193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.729761 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5"] Nov 03 16:18:35.734173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.733885 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.760758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.760106 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5"] Nov 03 16:18:35.790975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.790944 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knwt\" (UniqueName: \"kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.791124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.790995 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.791124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.791039 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.892193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.892153 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.892473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.892221 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knwt\" (UniqueName: \"kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.892473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.892281 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.892776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.892741 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.893024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.893004 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:35.928165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:35.928052 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knwt\" (UniqueName: \"kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:36.056030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.055736 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:18:36.122464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.122404 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2"] Nov 03 16:18:36.127914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.127887 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.153586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.150282 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9tjmt\"" Nov 03 16:18:36.153586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.150397 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 16:18:36.153586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.150507 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 16:18:36.153586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.150623 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 16:18:36.154751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.154726 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2"] Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200684 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200722 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200741 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9ww\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200772 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200788 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200801 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200839 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200858 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.201022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.200872 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.226882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.226802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:36.253454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.253261 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z"] Nov 03 16:18:36.257635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.256947 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.300415 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.300384 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z"] Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.301947 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.301990 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302022 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302054 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9ww\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302076 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302103 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302132 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302157 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302191 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4hb\" (UniqueName: \"kubernetes.io/projected/055bb54a-1e09-47bf-930a-1e90666cb38c-kube-api-access-bd4hb\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302240 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302264 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302286 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302319 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302362 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055bb54a-1e09-47bf-930a-1e90666cb38c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.302495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.302407 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.303764 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.303816 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.303864 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.304202 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.304722 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.304735 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.304967 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.306693 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.306652 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.314888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.314861 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.328604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.328555 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.334394 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.334344 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.336985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.336956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9ww\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-g2ss2\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.344054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.344032 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-nknlz\"" Nov 03 16:18:36.405281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405245 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5"] Nov 03 16:18:36.405699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405579 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405628 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055bb54a-1e09-47bf-930a-1e90666cb38c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405689 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405732 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405768 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405798 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405829 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405855 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.405929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405891 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4hb\" (UniqueName: \"kubernetes.io/projected/055bb54a-1e09-47bf-930a-1e90666cb38c-kube-api-access-bd4hb\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.406027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.405957 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.406226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.406201 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.407050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.406449 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.407050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.406683 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.407050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.406815 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055bb54a-1e09-47bf-930a-1e90666cb38c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.418694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.418562 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:36.420525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.420498 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.422582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.422482 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.432782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.432751 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4hb\" (UniqueName: \"kubernetes.io/projected/055bb54a-1e09-47bf-930a-1e90666cb38c-kube-api-access-bd4hb\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.441058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.441032 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055bb54a-1e09-47bf-930a-1e90666cb38c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z\" (UID: \"055bb54a-1e09-47bf-930a-1e90666cb38c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.467911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.467885 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:36.473598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.473569 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" event={"ID":"04a857a8-d0ba-4d3d-b48a-659f052f70bd","Type":"ContainerStarted","Data":"705ee27d4431564e6d3d5a5e8c8ad9803a12418668bcd569ca74d4f854b0cc28"} Nov 03 16:18:36.575199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.574857 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:36.821542 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.821230 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2"] Nov 03 16:18:36.824652 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:18:36.824622 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e8a12c_f0a1_4690_ab57_56958719f4ba.slice/crio-530c96e1519949c5aeb46dd7a1db27fdd7fb40bc1e08c319346cefcf56ef3aab WatchSource:0}: Error finding container 530c96e1519949c5aeb46dd7a1db27fdd7fb40bc1e08c319346cefcf56ef3aab: Status 404 returned error can't find the container with id 530c96e1519949c5aeb46dd7a1db27fdd7fb40bc1e08c319346cefcf56ef3aab Nov 03 16:18:36.923643 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:36.923552 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z"] Nov 03 16:18:37.500756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:37.500720 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" event={"ID":"40e8a12c-f0a1-4690-ab57-56958719f4ba","Type":"ContainerStarted","Data":"530c96e1519949c5aeb46dd7a1db27fdd7fb40bc1e08c319346cefcf56ef3aab"} Nov 03 16:18:37.504191 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:37.504055 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" event={"ID":"055bb54a-1e09-47bf-930a-1e90666cb38c","Type":"ContainerStarted","Data":"e9af386dfbc270e8c9b752381ae3891f2d020d31efd4d007b5b54cae1d9c5b96"} Nov 03 16:18:38.028193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:38.028156 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:38.186768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:38.186691 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:40.452817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:40.452725 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:43.586078 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:43.586038 4539 generic.go:356] "Generic (PLEG): container finished" podID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerID="f660ccf3b26b02bd5d44db5f10184b8e18070892bfcf07c7bcf0239ccb26a498" exitCode=0 Nov 03 16:18:43.586570 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:43.586138 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" event={"ID":"468ffc7f-a01a-4257-9bd5-59b361ad3e5c","Type":"ContainerDied","Data":"f660ccf3b26b02bd5d44db5f10184b8e18070892bfcf07c7bcf0239ccb26a498"} Nov 03 16:18:44.599757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:44.599716 4539 generic.go:356] "Generic (PLEG): container finished" podID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerID="dd645d196001018d661bbc9099e9766e700705f046910d5b86c56d7710287bba" exitCode=0 Nov 03 16:18:44.600375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:44.599772 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" event={"ID":"13d2adb5-b2e4-4c8e-90c1-bce49246cea9","Type":"ContainerDied","Data":"dd645d196001018d661bbc9099e9766e700705f046910d5b86c56d7710287bba"} Nov 03 16:18:45.297372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.297340 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:18:45.297488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.297440 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:18:45.297488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.297481 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:18:45.535517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.535473 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:18:45.535653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.535575 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:18:45.535653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.535622 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:18:45.627820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.627788 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" event={"ID":"055bb54a-1e09-47bf-930a-1e90666cb38c","Type":"ContainerStarted","Data":"8c7a9dd113fb1d9c8d6ce6125ecb98f6cf2e94636b1be5c00c803b26ae2cf805"} Nov 03 16:18:45.635602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.635573 4539 generic.go:356] "Generic (PLEG): container finished" podID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerID="406dc679f262d388f925c0266ad82f71e40517d26c611e705e87fde57d173b79" exitCode=0 Nov 03 16:18:45.635719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.635643 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" event={"ID":"13d2adb5-b2e4-4c8e-90c1-bce49246cea9","Type":"ContainerDied","Data":"406dc679f262d388f925c0266ad82f71e40517d26c611e705e87fde57d173b79"} Nov 03 16:18:45.684182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:45.682787 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" podStartSLOduration=1.3316440059999999 podStartE2EDuration="9.682769438s" totalImagesPullingTime="8.351125432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:45.679000725 +0000 UTC m=+269.372271745" watchObservedRunningTime="2026-11-03 16:18:45.682769438 +0000 UTC m=+269.376040460" Nov 03 16:18:46.575933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.575860 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:46.582507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.582237 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:46.642860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.641982 4539 generic.go:356] "Generic (PLEG): container finished" podID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerID="93967a34217206ed5a7c084a2bae528ec41ade74095b4c714732b015382c68fe" exitCode=0 Nov 03 16:18:46.642860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.642053 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerDied","Data":"93967a34217206ed5a7c084a2bae528ec41ade74095b4c714732b015382c68fe"} Nov 03 16:18:46.649163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.648351 4539 generic.go:356] "Generic (PLEG): container finished" podID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerID="efeb3128dff91f35c3b5a7690e2e17cb3701f5af2d37d603e5ae04e8fe8264ac" exitCode=0 Nov 03 16:18:46.649163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.648442 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" event={"ID":"f0f5320d-e122-4c82-b47d-7e956dbe7579","Type":"ContainerDied","Data":"efeb3128dff91f35c3b5a7690e2e17cb3701f5af2d37d603e5ae04e8fe8264ac"} Nov 03 16:18:46.658474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.658331 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" event={"ID":"40e8a12c-f0a1-4690-ab57-56958719f4ba","Type":"ContainerStarted","Data":"72ff845877700413c53a4664fc691d7d224f85211bebb9adc16b42ad9dc505f2"} Nov 03 16:18:46.666470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.666400 4539 generic.go:356] "Generic (PLEG): container finished" podID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerID="57a10d7bde9bf1576e7849290dab25c18321c6df7422f594d2f89be208d5aa95" exitCode=0 Nov 03 16:18:46.666576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.666506 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" event={"ID":"e52e0c92-764f-49b9-9d87-879fa5c094ff","Type":"ContainerDied","Data":"57a10d7bde9bf1576e7849290dab25c18321c6df7422f594d2f89be208d5aa95"} Nov 03 16:18:46.672546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.672522 4539 generic.go:356] "Generic (PLEG): container finished" podID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerID="bf089667739d4b23e95e355c902413c2ae7298589564630a1fed8dea1050ea54" exitCode=0 Nov 03 16:18:46.673527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.673481 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" event={"ID":"c25e89b0-49eb-419a-9682-0f9b1143e0a2","Type":"ContainerDied","Data":"bf089667739d4b23e95e355c902413c2ae7298589564630a1fed8dea1050ea54"} Nov 03 16:18:46.673600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.673529 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:46.674335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.674319 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z" Nov 03 16:18:46.812842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:46.812788 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podStartSLOduration=2.106689407 podStartE2EDuration="10.81276981s" totalImagesPullingTime="8.706080403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:46.811205691 +0000 UTC m=+270.504476714" watchObservedRunningTime="2026-11-03 16:18:46.81276981 +0000 UTC m=+270.506040830" Nov 03 16:18:47.468634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.468599 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:47.473664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.473637 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:47.680056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.679973 4539 generic.go:356] "Generic (PLEG): container finished" podID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerID="768f96925abdb8848ad64871201b6782925a84358861998926cd9414d486bcfa" exitCode=0 Nov 03 16:18:47.680504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.680478 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" event={"ID":"04a857a8-d0ba-4d3d-b48a-659f052f70bd","Type":"ContainerDied","Data":"768f96925abdb8848ad64871201b6782925a84358861998926cd9414d486bcfa"} Nov 03 16:18:47.682992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.682968 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" event={"ID":"468ffc7f-a01a-4257-9bd5-59b361ad3e5c","Type":"ContainerStarted","Data":"dce350764ea18da6153d859c287c6d6d975926e8a644edbab0ec0ec5d0806baa"} Nov 03 16:18:47.685262 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.685235 4539 generic.go:356] "Generic (PLEG): container finished" podID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerID="75e4c6320dd43055bb64b2a36076c687dd61a06e405ff3b7009ad7e1d083508c" exitCode=0 Nov 03 16:18:47.685389 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.685368 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" event={"ID":"8c58a485-0bca-423c-bf35-4853fb9fd674","Type":"ContainerDied","Data":"75e4c6320dd43055bb64b2a36076c687dd61a06e405ff3b7009ad7e1d083508c"} Nov 03 16:18:47.687768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.687733 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerStarted","Data":"d4a1155b93f74c19637a8510649765e3fa708fd6519277c4fa811ef2c2ae4865"} Nov 03 16:18:47.692940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.692917 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" event={"ID":"f0f5320d-e122-4c82-b47d-7e956dbe7579","Type":"ContainerStarted","Data":"2e2c817b0087ecaa8255ec5c640d08555884be0f6ed2338ac933cc11343dab81"} Nov 03 16:18:47.693367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.693351 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:47.694491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:47.694472 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:18:48.702408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.702369 4539 generic.go:356] "Generic (PLEG): container finished" podID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerID="d4a1155b93f74c19637a8510649765e3fa708fd6519277c4fa811ef2c2ae4865" exitCode=0 Nov 03 16:18:48.702837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.702600 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerDied","Data":"d4a1155b93f74c19637a8510649765e3fa708fd6519277c4fa811ef2c2ae4865"} Nov 03 16:18:48.705555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.705526 4539 generic.go:356] "Generic (PLEG): container finished" podID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerID="2e2c817b0087ecaa8255ec5c640d08555884be0f6ed2338ac933cc11343dab81" exitCode=0 Nov 03 16:18:48.705682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.705558 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" event={"ID":"f0f5320d-e122-4c82-b47d-7e956dbe7579","Type":"ContainerDied","Data":"2e2c817b0087ecaa8255ec5c640d08555884be0f6ed2338ac933cc11343dab81"} Nov 03 16:18:48.708780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.708757 4539 generic.go:356] "Generic (PLEG): container finished" podID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerID="f8fa31620c50affa3bf0a64757092257a48d4673181ac4cf5f8fd159a557948c" exitCode=0 Nov 03 16:18:48.708908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.708825 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" event={"ID":"e52e0c92-764f-49b9-9d87-879fa5c094ff","Type":"ContainerDied","Data":"f8fa31620c50affa3bf0a64757092257a48d4673181ac4cf5f8fd159a557948c"} Nov 03 16:18:48.713297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.713276 4539 generic.go:356] "Generic (PLEG): container finished" podID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerID="cc15626c01006b5bad749ca967c7aa958019788eacc715fe09f924d22148c2b5" exitCode=0 Nov 03 16:18:48.713482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.713459 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" event={"ID":"c25e89b0-49eb-419a-9682-0f9b1143e0a2","Type":"ContainerDied","Data":"cc15626c01006b5bad749ca967c7aa958019788eacc715fe09f924d22148c2b5"} Nov 03 16:18:48.719820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.719719 4539 generic.go:356] "Generic (PLEG): container finished" podID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerID="7d1873775b3bf4bdca0b6b4da3f32c753933aca8c03a112573a56d8991ebaa69" exitCode=0 Nov 03 16:18:48.719820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.719796 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" event={"ID":"04a857a8-d0ba-4d3d-b48a-659f052f70bd","Type":"ContainerDied","Data":"7d1873775b3bf4bdca0b6b4da3f32c753933aca8c03a112573a56d8991ebaa69"} Nov 03 16:18:48.724987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.724964 4539 generic.go:356] "Generic (PLEG): container finished" podID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerID="dce350764ea18da6153d859c287c6d6d975926e8a644edbab0ec0ec5d0806baa" exitCode=0 Nov 03 16:18:48.726025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:48.726004 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" event={"ID":"468ffc7f-a01a-4257-9bd5-59b361ad3e5c","Type":"ContainerDied","Data":"dce350764ea18da6153d859c287c6d6d975926e8a644edbab0ec0ec5d0806baa"} Nov 03 16:18:53.782604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:53.782437 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" event={"ID":"8c58a485-0bca-423c-bf35-4853fb9fd674","Type":"ContainerDied","Data":"a7ccbbdbce63285f6e544f69242a37f20ef4667bd9027f317039a7018e912463"} Nov 03 16:18:53.783067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:53.782688 4539 generic.go:356] "Generic (PLEG): container finished" podID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerID="a7ccbbdbce63285f6e544f69242a37f20ef4667bd9027f317039a7018e912463" exitCode=0 Nov 03 16:18:56.188148 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:18:56.188062 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:18:59.176410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:59.176361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:59.381416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:59.381382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:18:59.765054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:18:59.765018 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:00.125749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.125711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:00.290244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.290215 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd"] Nov 03 16:19:00.313708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.313674 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.346521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.346473 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd"] Nov 03 16:19:00.358642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.358605 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 16:19:00.358642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.358636 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 16:19:00.358868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.358751 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 16:19:00.358868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.358758 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-k6tn8\"" Nov 03 16:19:00.383110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.383037 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 16:19:00.383281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.383137 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 16:19:00.392803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.392766 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:00.451703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.451648 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.451891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.451710 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.451891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.451795 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.451891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.451853 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.452089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.451954 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.452089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.452024 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f44\" (UniqueName: \"kubernetes.io/projected/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-kube-api-access-s8f44\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.553597 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.553661 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.553716 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f44\" (UniqueName: \"kubernetes.io/projected/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-kube-api-access-s8f44\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.553771 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.553824 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.553850 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.554564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.554298 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.558953 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.558843 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.558953 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.558981 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.559284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.559084 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.562883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.562831 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.582705 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.582626 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:00.588306 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.588242 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f44\" (UniqueName: \"kubernetes.io/projected/8360e90b-7b15-4a64-9fe0-f5d2acaa6403-kube-api-access-s8f44\") pod \"opendatahub-operator-controller-manager-698cf769bf-smbdd\" (UID: \"8360e90b-7b15-4a64-9fe0-f5d2acaa6403\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.663986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.663694 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:00.762841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.762745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:00.878848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.878320 4539 generic.go:356] "Generic (PLEG): container finished" podID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerID="d244c4a50be36ef70a79f01bf7b219edbfe406db47834299b610e516db8f45ca" exitCode=0 Nov 03 16:19:00.878848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.878453 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" event={"ID":"13d2adb5-b2e4-4c8e-90c1-bce49246cea9","Type":"ContainerDied","Data":"d244c4a50be36ef70a79f01bf7b219edbfe406db47834299b610e516db8f45ca"} Nov 03 16:19:00.942211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:00.942075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:01.090796 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:01.090771 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd"] Nov 03 16:19:01.094071 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:01.094035 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8360e90b_7b15_4a64_9fe0_f5d2acaa6403.slice/crio-eac72839908cb8aceab923b3a5e0907c2354b42b8b8f76641540b2c34c51a2cd WatchSource:0}: Error finding container eac72839908cb8aceab923b3a5e0907c2354b42b8b8f76641540b2c34c51a2cd: Status 404 returned error can't find the container with id eac72839908cb8aceab923b3a5e0907c2354b42b8b8f76641540b2c34c51a2cd Nov 03 16:19:01.894083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:01.894030 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" event={"ID":"8360e90b-7b15-4a64-9fe0-f5d2acaa6403","Type":"ContainerStarted","Data":"eac72839908cb8aceab923b3a5e0907c2354b42b8b8f76641540b2c34c51a2cd"} Nov 03 16:19:02.098907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.098879 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:19:02.269933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.269849 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5\" (UniqueName: \"kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5\") pod \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " Nov 03 16:19:02.269933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.269927 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util\") pod \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " Nov 03 16:19:02.270191 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.269994 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle\") pod \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\" (UID: \"13d2adb5-b2e4-4c8e-90c1-bce49246cea9\") " Nov 03 16:19:02.271509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.271478 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle" pod "13d2adb5-b2e4-4c8e-90c1-bce49246cea9" (UID: "13d2adb5-b2e4-4c8e-90c1-bce49246cea9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:02.277170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.277135 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5" pod "13d2adb5-b2e4-4c8e-90c1-bce49246cea9" (UID: "13d2adb5-b2e4-4c8e-90c1-bce49246cea9"). InnerVolumeSpecName "kube-api-access-vs7d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:02.277365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.277345 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util" pod "13d2adb5-b2e4-4c8e-90c1-bce49246cea9" (UID: "13d2adb5-b2e4-4c8e-90c1-bce49246cea9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:02.371038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.370964 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:02.371038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.371002 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:02.371038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.371017 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7d5\" (UniqueName: \"kubernetes.io/projected/13d2adb5-b2e4-4c8e-90c1-bce49246cea9-kube-api-access-vs7d5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:02.612752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.612714 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:02.795645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.795610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:02.908634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.908488 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" event={"ID":"13d2adb5-b2e4-4c8e-90c1-bce49246cea9","Type":"ContainerDied","Data":"16ef69e0520fc528f0a8c525ac9e2c41047b6f3fc4d141c72e3342aac5eb2c9c"} Nov 03 16:19:02.908634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.908561 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ef69e0520fc528f0a8c525ac9e2c41047b6f3fc4d141c72e3342aac5eb2c9c" Nov 03 16:19:02.908634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:02.908622 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bz5gt" Nov 03 16:19:04.938718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:04.938681 4539 generic.go:356] "Generic (PLEG): container finished" podID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerID="09c65dc034c55df2737b932be70666fcc92a715db2b3a481dbd8a67e62b83ecb" exitCode=0 Nov 03 16:19:04.939190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:04.938776 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" event={"ID":"8c58a485-0bca-423c-bf35-4853fb9fd674","Type":"ContainerDied","Data":"09c65dc034c55df2737b932be70666fcc92a715db2b3a481dbd8a67e62b83ecb"} Nov 03 16:19:04.945686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:04.945648 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerStarted","Data":"ee7cfc36f4dfbb90003d8eb808e15de7614d5a017c92811646cecd0f3698061f"} Nov 03 16:19:04.950012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:04.949977 4539 generic.go:356] "Generic (PLEG): container finished" podID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerID="aeb76e878448ae796860e507cab20b29fe7710dd6e391e0736487d01e831815c" exitCode=0 Nov 03 16:19:04.950167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:04.950043 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" event={"ID":"f0f5320d-e122-4c82-b47d-7e956dbe7579","Type":"ContainerDied","Data":"aeb76e878448ae796860e507cab20b29fe7710dd6e391e0736487d01e831815c"} Nov 03 16:19:05.957494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:05.957278 4539 generic.go:356] "Generic (PLEG): container finished" podID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerID="ee7cfc36f4dfbb90003d8eb808e15de7614d5a017c92811646cecd0f3698061f" exitCode=0 Nov 03 16:19:05.957494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:05.957363 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerDied","Data":"ee7cfc36f4dfbb90003d8eb808e15de7614d5a017c92811646cecd0f3698061f"} Nov 03 16:19:05.960381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:05.960354 4539 generic.go:356] "Generic (PLEG): container finished" podID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerID="c5dc8c0d24c18ebb2bb1c8d435df75a4a22947346da02ea81c13df7ee7b14010" exitCode=0 Nov 03 16:19:05.960541 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:05.960463 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" event={"ID":"04a857a8-d0ba-4d3d-b48a-659f052f70bd","Type":"ContainerDied","Data":"c5dc8c0d24c18ebb2bb1c8d435df75a4a22947346da02ea81c13df7ee7b14010"} Nov 03 16:19:06.158468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.158412 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:19:06.160086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.160055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:06.173743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.173721 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:19:06.311865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.311831 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util\") pod \"f0f5320d-e122-4c82-b47d-7e956dbe7579\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " Nov 03 16:19:06.312028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.311908 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle\") pod \"8c58a485-0bca-423c-bf35-4853fb9fd674\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " Nov 03 16:19:06.312028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.311940 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx\" (UniqueName: \"kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx\") pod \"f0f5320d-e122-4c82-b47d-7e956dbe7579\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " Nov 03 16:19:06.312028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.311988 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle\") pod \"f0f5320d-e122-4c82-b47d-7e956dbe7579\" (UID: \"f0f5320d-e122-4c82-b47d-7e956dbe7579\") " Nov 03 16:19:06.312028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.312019 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util\") pod \"8c58a485-0bca-423c-bf35-4853fb9fd674\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " Nov 03 16:19:06.312193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.312063 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz\" (UniqueName: \"kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz\") pod \"8c58a485-0bca-423c-bf35-4853fb9fd674\" (UID: \"8c58a485-0bca-423c-bf35-4853fb9fd674\") " Nov 03 16:19:06.312844 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.312739 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle" pod "f0f5320d-e122-4c82-b47d-7e956dbe7579" (UID: "f0f5320d-e122-4c82-b47d-7e956dbe7579"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:06.315196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.315170 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz" pod "8c58a485-0bca-423c-bf35-4853fb9fd674" (UID: "8c58a485-0bca-423c-bf35-4853fb9fd674"). InnerVolumeSpecName "kube-api-access-cz5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:06.315309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.315216 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx" pod "f0f5320d-e122-4c82-b47d-7e956dbe7579" (UID: "f0f5320d-e122-4c82-b47d-7e956dbe7579"). InnerVolumeSpecName "kube-api-access-xgpnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:06.315346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.315305 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle" pod "8c58a485-0bca-423c-bf35-4853fb9fd674" (UID: "8c58a485-0bca-423c-bf35-4853fb9fd674"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:06.317486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.317467 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util" pod "f0f5320d-e122-4c82-b47d-7e956dbe7579" (UID: "f0f5320d-e122-4c82-b47d-7e956dbe7579"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:06.321293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.321270 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util" pod "8c58a485-0bca-423c-bf35-4853fb9fd674" (UID: "8c58a485-0bca-423c-bf35-4853fb9fd674"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:06.335797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.335775 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:06.412939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.412884 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:06.412939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.412904 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5rz\" (UniqueName: \"kubernetes.io/projected/8c58a485-0bca-423c-bf35-4853fb9fd674-kube-api-access-cz5rz\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:06.412939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.412914 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:06.412939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.412924 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c58a485-0bca-423c-bf35-4853fb9fd674-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:06.412939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.412936 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpnx\" (UniqueName: \"kubernetes.io/projected/f0f5320d-e122-4c82-b47d-7e956dbe7579-kube-api-access-xgpnx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:06.413204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.412945 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f5320d-e122-4c82-b47d-7e956dbe7579-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:06.942988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.942949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:06.966417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.966337 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" event={"ID":"8360e90b-7b15-4a64-9fe0-f5d2acaa6403","Type":"ContainerStarted","Data":"6ccf46974f67362c28c84c63c0de6f3fbefd9607bc6acc2c39ab51a96ab8e2dc"} Nov 03 16:19:06.968359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.968334 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" event={"ID":"8c58a485-0bca-423c-bf35-4853fb9fd674","Type":"ContainerDied","Data":"047bec9ae438010ce95fbfcd7e7de04c05bf5b6c943a12e71c3ed4f0dd013957"} Nov 03 16:19:06.968484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.968384 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="047bec9ae438010ce95fbfcd7e7de04c05bf5b6c943a12e71c3ed4f0dd013957" Nov 03 16:19:06.968484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.968346 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447phlc" Nov 03 16:19:06.970344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.970319 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" Nov 03 16:19:06.970466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.970358 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hch5f" event={"ID":"f0f5320d-e122-4c82-b47d-7e956dbe7579","Type":"ContainerDied","Data":"dabef216a9b15ab3def09b26c637af6294b09ac22f869c81185c052ed45c153e"} Nov 03 16:19:06.970466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.970389 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dabef216a9b15ab3def09b26c637af6294b09ac22f869c81185c052ed45c153e" Nov 03 16:19:06.972927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.972908 4539 generic.go:356] "Generic (PLEG): container finished" podID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerID="a2c9a5a392957e3806eadcfda239a8ce6a14e774f19f6c8067fad9c3c760f663" exitCode=0 Nov 03 16:19:06.973024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.972985 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" event={"ID":"e52e0c92-764f-49b9-9d87-879fa5c094ff","Type":"ContainerDied","Data":"a2c9a5a392957e3806eadcfda239a8ce6a14e774f19f6c8067fad9c3c760f663"} Nov 03 16:19:06.975088 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.975067 4539 generic.go:356] "Generic (PLEG): container finished" podID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerID="daa114121409dee29e67683c0fb8dd185ef0abd1a060aa2bed445bcb9713d293" exitCode=0 Nov 03 16:19:06.975176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.975133 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" event={"ID":"c25e89b0-49eb-419a-9682-0f9b1143e0a2","Type":"ContainerDied","Data":"daa114121409dee29e67683c0fb8dd185ef0abd1a060aa2bed445bcb9713d293"} Nov 03 16:19:06.977208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.977188 4539 generic.go:356] "Generic (PLEG): container finished" podID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerID="242079940c12aee64f8f4bb0abeac747ad29ff3a20117af1d66c91392ab65458" exitCode=0 Nov 03 16:19:06.977315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:06.977263 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" event={"ID":"468ffc7f-a01a-4257-9bd5-59b361ad3e5c","Type":"ContainerDied","Data":"242079940c12aee64f8f4bb0abeac747ad29ff3a20117af1d66c91392ab65458"} Nov 03 16:19:07.115617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.115592 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:19:07.117763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.117746 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:19:07.218823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.218752 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq\" (UniqueName: \"kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq\") pod \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " Nov 03 16:19:07.218823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.218787 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle\") pod \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " Nov 03 16:19:07.219018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.218832 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util\") pod \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " Nov 03 16:19:07.219018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.218862 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt\" (UniqueName: \"kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt\") pod \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " Nov 03 16:19:07.219018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.218886 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util\") pod \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\" (UID: \"61745b9c-ec5b-42c8-8a05-03c036cb48a0\") " Nov 03 16:19:07.219018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.218934 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle\") pod \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\" (UID: \"04a857a8-d0ba-4d3d-b48a-659f052f70bd\") " Nov 03 16:19:07.219620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.219594 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle" pod "61745b9c-ec5b-42c8-8a05-03c036cb48a0" (UID: "61745b9c-ec5b-42c8-8a05-03c036cb48a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:07.219759 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.219732 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle" pod "04a857a8-d0ba-4d3d-b48a-659f052f70bd" (UID: "04a857a8-d0ba-4d3d-b48a-659f052f70bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:07.221390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.221367 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq" pod "61745b9c-ec5b-42c8-8a05-03c036cb48a0" (UID: "61745b9c-ec5b-42c8-8a05-03c036cb48a0"). InnerVolumeSpecName "kube-api-access-fbbxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:07.221390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.221378 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt" pod "04a857a8-d0ba-4d3d-b48a-659f052f70bd" (UID: "04a857a8-d0ba-4d3d-b48a-659f052f70bd"). InnerVolumeSpecName "kube-api-access-2knwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:07.223945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.223926 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util" pod "04a857a8-d0ba-4d3d-b48a-659f052f70bd" (UID: "04a857a8-d0ba-4d3d-b48a-659f052f70bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:07.225209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.225189 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util" pod "61745b9c-ec5b-42c8-8a05-03c036cb48a0" (UID: "61745b9c-ec5b-42c8-8a05-03c036cb48a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:07.320230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.320174 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:07.320230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.320220 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbxq\" (UniqueName: \"kubernetes.io/projected/61745b9c-ec5b-42c8-8a05-03c036cb48a0-kube-api-access-fbbxq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:07.320230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.320235 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:07.320489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.320246 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04a857a8-d0ba-4d3d-b48a-659f052f70bd-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:07.320489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.320258 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knwt\" (UniqueName: \"kubernetes.io/projected/04a857a8-d0ba-4d3d-b48a-659f052f70bd-kube-api-access-2knwt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:07.320489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.320267 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61745b9c-ec5b-42c8-8a05-03c036cb48a0-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:07.983507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.983474 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" Nov 03 16:19:07.983507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.983487 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sqk5" event={"ID":"04a857a8-d0ba-4d3d-b48a-659f052f70bd","Type":"ContainerDied","Data":"705ee27d4431564e6d3d5a5e8c8ad9803a12418668bcd569ca74d4f854b0cc28"} Nov 03 16:19:07.983948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.983533 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705ee27d4431564e6d3d5a5e8c8ad9803a12418668bcd569ca74d4f854b0cc28" Nov 03 16:19:07.985187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.985165 4539 generic.go:356] "Generic (PLEG): container finished" podID="8360e90b-7b15-4a64-9fe0-f5d2acaa6403" containerID="6ccf46974f67362c28c84c63c0de6f3fbefd9607bc6acc2c39ab51a96ab8e2dc" exitCode=0 Nov 03 16:19:07.985300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.985260 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" event={"ID":"8360e90b-7b15-4a64-9fe0-f5d2acaa6403","Type":"ContainerDied","Data":"6ccf46974f67362c28c84c63c0de6f3fbefd9607bc6acc2c39ab51a96ab8e2dc"} Nov 03 16:19:07.987213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.987188 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" event={"ID":"61745b9c-ec5b-42c8-8a05-03c036cb48a0","Type":"ContainerDied","Data":"bbf78c5269744af51e3235da6dc60b884a2729eedfca1ebe504702126f27b0c8"} Nov 03 16:19:07.987287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.987231 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkqkz" Nov 03 16:19:07.987287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:07.987245 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbf78c5269744af51e3235da6dc60b884a2729eedfca1ebe504702126f27b0c8" Nov 03 16:19:08.118554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.118531 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:19:08.174087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.174054 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:19:08.177318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.177296 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:19:08.230234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.230206 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util\") pod \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " Nov 03 16:19:08.230451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.230314 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle\") pod \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " Nov 03 16:19:08.230451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.230352 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct\" (UniqueName: \"kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct\") pod \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\" (UID: \"468ffc7f-a01a-4257-9bd5-59b361ad3e5c\") " Nov 03 16:19:08.231050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.231020 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle" pod "468ffc7f-a01a-4257-9bd5-59b361ad3e5c" (UID: "468ffc7f-a01a-4257-9bd5-59b361ad3e5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:08.232609 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.232586 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct" pod "468ffc7f-a01a-4257-9bd5-59b361ad3e5c" (UID: "468ffc7f-a01a-4257-9bd5-59b361ad3e5c"). InnerVolumeSpecName "kube-api-access-sh2ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:08.235762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.235714 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util" pod "468ffc7f-a01a-4257-9bd5-59b361ad3e5c" (UID: "468ffc7f-a01a-4257-9bd5-59b361ad3e5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:08.330998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.330973 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util\") pod \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " Nov 03 16:19:08.331105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331090 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util\") pod \"e52e0c92-764f-49b9-9d87-879fa5c094ff\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " Nov 03 16:19:08.331172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331131 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle\") pod \"e52e0c92-764f-49b9-9d87-879fa5c094ff\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " Nov 03 16:19:08.331238 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331222 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx\" (UniqueName: \"kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx\") pod \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " Nov 03 16:19:08.331297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331256 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk\" (UniqueName: \"kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk\") pod \"e52e0c92-764f-49b9-9d87-879fa5c094ff\" (UID: \"e52e0c92-764f-49b9-9d87-879fa5c094ff\") " Nov 03 16:19:08.331297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331289 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle\") pod \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\" (UID: \"c25e89b0-49eb-419a-9682-0f9b1143e0a2\") " Nov 03 16:19:08.331922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331492 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.331922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331514 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2ct\" (UniqueName: \"kubernetes.io/projected/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-kube-api-access-sh2ct\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.331922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331529 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/468ffc7f-a01a-4257-9bd5-59b361ad3e5c-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.331922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331842 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle" pod "e52e0c92-764f-49b9-9d87-879fa5c094ff" (UID: "e52e0c92-764f-49b9-9d87-879fa5c094ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:08.332110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.331954 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle" pod "c25e89b0-49eb-419a-9682-0f9b1143e0a2" (UID: "c25e89b0-49eb-419a-9682-0f9b1143e0a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:08.333716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.333665 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk" pod "e52e0c92-764f-49b9-9d87-879fa5c094ff" (UID: "e52e0c92-764f-49b9-9d87-879fa5c094ff"). InnerVolumeSpecName "kube-api-access-g9bkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:08.333822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.333791 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx" pod "c25e89b0-49eb-419a-9682-0f9b1143e0a2" (UID: "c25e89b0-49eb-419a-9682-0f9b1143e0a2"). InnerVolumeSpecName "kube-api-access-rslmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:19:08.337177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.337154 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util" pod "e52e0c92-764f-49b9-9d87-879fa5c094ff" (UID: "e52e0c92-764f-49b9-9d87-879fa5c094ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:08.337744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.337723 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util" pod "c25e89b0-49eb-419a-9682-0f9b1143e0a2" (UID: "c25e89b0-49eb-419a-9682-0f9b1143e0a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:19:08.432079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.432042 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rslmx\" (UniqueName: \"kubernetes.io/projected/c25e89b0-49eb-419a-9682-0f9b1143e0a2-kube-api-access-rslmx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.432079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.432071 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bkk\" (UniqueName: \"kubernetes.io/projected/e52e0c92-764f-49b9-9d87-879fa5c094ff-kube-api-access-g9bkk\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.432079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.432080 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.432079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.432090 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c25e89b0-49eb-419a-9682-0f9b1143e0a2-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.432356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.432099 4539 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-util\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.432356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.432107 4539 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e52e0c92-764f-49b9-9d87-879fa5c094ff-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:19:08.992921 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.992892 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" Nov 03 16:19:08.993366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.992889 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f9zb7" event={"ID":"e52e0c92-764f-49b9-9d87-879fa5c094ff","Type":"ContainerDied","Data":"ce264582be253c8e4cc98c614fd3d0a1f7d19f47211c1ab03f7d4cf1d32f5e1a"} Nov 03 16:19:08.993366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.993022 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce264582be253c8e4cc98c614fd3d0a1f7d19f47211c1ab03f7d4cf1d32f5e1a" Nov 03 16:19:08.994887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.994851 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" Nov 03 16:19:08.994887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.994858 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4s8xc" event={"ID":"c25e89b0-49eb-419a-9682-0f9b1143e0a2","Type":"ContainerDied","Data":"15ab333b2ac6373ed1ab487775b5d293a3f51e3703e0a24f23cea85cd5a9c725"} Nov 03 16:19:08.995083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.994898 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15ab333b2ac6373ed1ab487775b5d293a3f51e3703e0a24f23cea85cd5a9c725" Nov 03 16:19:08.997145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.997117 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" event={"ID":"468ffc7f-a01a-4257-9bd5-59b361ad3e5c","Type":"ContainerDied","Data":"6fecb72ba141aad4b806f741bfe9857696a928c92165c9473a64b937851f7868"} Nov 03 16:19:08.997278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.997159 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fecb72ba141aad4b806f741bfe9857696a928c92165c9473a64b937851f7868" Nov 03 16:19:08.997278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.997237 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cff2qj" Nov 03 16:19:08.999271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:08.999249 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" event={"ID":"8360e90b-7b15-4a64-9fe0-f5d2acaa6403","Type":"ContainerStarted","Data":"a40c754da42b07bca117ce5cfda7e47ea5bfd5a2a458ad9bced40519f499cb67"} Nov 03 16:19:09.127986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:09.127940 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" podStartSLOduration=2.607420546 podStartE2EDuration="9.127924393s" totalImagesPullingTime="5.980661812s" totalInitContainerRuntime="539.842035ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:09.117269316 +0000 UTC m=+292.810540337" watchObservedRunningTime="2026-11-03 16:19:09.127924393 +0000 UTC m=+292.821195415" Nov 03 16:19:10.004353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:10.004321 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:12.535752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:12.535706 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:12.700296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:12.700263 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:16.474958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:16.474932 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:19:16.475448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:16.475091 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:19:16.770040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:16.769962 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:20.249025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.248989 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2"] Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249384 4539 cpu_manager.go:414] "containerMap: removing container" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="pull" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249403 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="pull" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249412 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="extract" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249418 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="extract" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249446 4539 cpu_manager.go:414] "containerMap: removing container" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="pull" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249453 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="pull" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249460 4539 cpu_manager.go:414] "containerMap: removing container" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="extract" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249466 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="extract" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249476 4539 cpu_manager.go:414] "containerMap: removing container" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="extract" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249483 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="extract" Nov 03 16:19:20.249490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249492 4539 cpu_manager.go:414] "containerMap: removing container" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249496 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249503 4539 cpu_manager.go:414] "containerMap: removing container" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249508 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249517 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249521 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249528 4539 cpu_manager.go:414] "containerMap: removing container" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249532 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249538 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249542 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249551 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249558 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249567 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249570 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249576 4539 cpu_manager.go:414] "containerMap: removing container" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249580 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249585 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249600 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249607 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249612 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249618 4539 cpu_manager.go:414] "containerMap: removing container" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249624 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249634 4539 cpu_manager.go:414] "containerMap: removing container" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249643 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249650 4539 cpu_manager.go:414] "containerMap: removing container" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249654 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249661 4539 cpu_manager.go:414] "containerMap: removing container" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249665 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249672 4539 cpu_manager.go:414] "containerMap: removing container" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249678 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249684 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249689 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249695 4539 cpu_manager.go:414] "containerMap: removing container" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249699 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249707 4539 cpu_manager.go:414] "containerMap: removing container" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249714 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="util" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249724 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249729 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="pull" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249779 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25e89b0-49eb-419a-9682-0f9b1143e0a2" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249787 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d2adb5-b2e4-4c8e-90c1-bce49246cea9" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249796 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="468ffc7f-a01a-4257-9bd5-59b361ad3e5c" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249806 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a857a8-d0ba-4d3d-b48a-659f052f70bd" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249812 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c58a485-0bca-423c-bf35-4853fb9fd674" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249817 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f5320d-e122-4c82-b47d-7e956dbe7579" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249822 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="61745b9c-ec5b-42c8-8a05-03c036cb48a0" containerName="extract" Nov 03 16:19:20.249820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.249827 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52e0c92-764f-49b9-9d87-879fa5c094ff" containerName="extract" Nov 03 16:19:20.255872 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.255857 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:20.280479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.280452 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 16:19:20.280610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.280587 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 16:19:20.280682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.280629 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-g545m\"" Nov 03 16:19:20.282349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.282328 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2"] Nov 03 16:19:20.323402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.323365 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6j5c\" (UniqueName: \"kubernetes.io/projected/6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b-kube-api-access-b6j5c\") pod \"limitador-operator-controller-manager-5778bb4449-5ldg2\" (UID: \"6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:20.423920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.423870 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6j5c\" (UniqueName: \"kubernetes.io/projected/6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b-kube-api-access-b6j5c\") pod \"limitador-operator-controller-manager-5778bb4449-5ldg2\" (UID: \"6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:20.449203 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.449179 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6j5c\" (UniqueName: \"kubernetes.io/projected/6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b-kube-api-access-b6j5c\") pod \"limitador-operator-controller-manager-5778bb4449-5ldg2\" (UID: \"6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:20.568382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.568351 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:20.869751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.869509 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2"] Nov 03 16:19:20.871644 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:20.871617 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfff5e2_954c_4ec8_87d0_5d3c61e33e3b.slice/crio-2be8be835b116be74c7a511a7da8da0b4f2353d2de70c887ab381a5a5540d963 WatchSource:0}: Error finding container 2be8be835b116be74c7a511a7da8da0b4f2353d2de70c887ab381a5a5540d963: Status 404 returned error can't find the container with id 2be8be835b116be74c7a511a7da8da0b4f2353d2de70c887ab381a5a5540d963 Nov 03 16:19:20.874511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.874487 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:19:20.984986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:20.984954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:21.012650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:21.012625 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-smbdd" Nov 03 16:19:21.052023 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:21.051990 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" event={"ID":"6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b","Type":"ContainerStarted","Data":"2be8be835b116be74c7a511a7da8da0b4f2353d2de70c887ab381a5a5540d963"} Nov 03 16:19:21.778036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:21.778004 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:24.068075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.068041 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" event={"ID":"6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b","Type":"ContainerStarted","Data":"70473e4240e7c55613e9b675c2f3264aa48e4d4282e0317dcd3810d19bbe8ae1"} Nov 03 16:19:24.068495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.068116 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:24.123286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.123208 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" podStartSLOduration=1.641208358 podStartE2EDuration="4.123193125s" totalImagesPullingTime="2.481984767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:24.122046453 +0000 UTC m=+307.815317474" watchObservedRunningTime="2026-11-03 16:19:24.123193125 +0000 UTC m=+307.816464144" Nov 03 16:19:24.880537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.880506 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs"] Nov 03 16:19:24.884154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.884123 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:24.909152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.909118 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hmjmr\"" Nov 03 16:19:24.920926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.920898 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs"] Nov 03 16:19:24.963418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:24.963376 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstdg\" (UniqueName: \"kubernetes.io/projected/dfbb04a9-f7d4-4438-9465-867b1f6ab446-kube-api-access-nstdg\") pod \"authorino-operator-7b7bb4bc77-mqxzs\" (UID: \"dfbb04a9-f7d4-4438-9465-867b1f6ab446\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:25.024410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:25.024373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:25.064410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:25.064368 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstdg\" (UniqueName: \"kubernetes.io/projected/dfbb04a9-f7d4-4438-9465-867b1f6ab446-kube-api-access-nstdg\") pod \"authorino-operator-7b7bb4bc77-mqxzs\" (UID: \"dfbb04a9-f7d4-4438-9465-867b1f6ab446\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:25.088947 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:25.088910 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstdg\" (UniqueName: \"kubernetes.io/projected/dfbb04a9-f7d4-4438-9465-867b1f6ab446-kube-api-access-nstdg\") pod \"authorino-operator-7b7bb4bc77-mqxzs\" (UID: \"dfbb04a9-f7d4-4438-9465-867b1f6ab446\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:25.197153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:25.197060 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:25.514722 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:25.514645 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs"] Nov 03 16:19:25.516884 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:25.516856 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbb04a9_f7d4_4438_9465_867b1f6ab446.slice/crio-8c3ec1043f241141e520636a7dfd1ce6fd4af53cd606e4ef0a8ad51313b6f744 WatchSource:0}: Error finding container 8c3ec1043f241141e520636a7dfd1ce6fd4af53cd606e4ef0a8ad51313b6f744: Status 404 returned error can't find the container with id 8c3ec1043f241141e520636a7dfd1ce6fd4af53cd606e4ef0a8ad51313b6f744 Nov 03 16:19:26.077649 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:26.077617 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" event={"ID":"dfbb04a9-f7d4-4438-9465-867b1f6ab446","Type":"ContainerStarted","Data":"8c3ec1043f241141e520636a7dfd1ce6fd4af53cd606e4ef0a8ad51313b6f744"} Nov 03 16:19:26.155169 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:19:26.155121 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:19:26.401278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:26.401200 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:27.760591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:27.760549 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:27.977201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:27.977165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:28.087697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:28.087663 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" event={"ID":"dfbb04a9-f7d4-4438-9465-867b1f6ab446","Type":"ContainerStarted","Data":"290ddb7fcca87abd4c5e28ef0c24d23a868e30e3851b44fb98dc9af0aa612988"} Nov 03 16:19:28.087874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:28.087811 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:28.180335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:28.180287 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" podStartSLOduration=2.593346362 podStartE2EDuration="4.180274335s" totalImagesPullingTime="1.586927973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:28.177360426 +0000 UTC m=+311.870631447" watchObservedRunningTime="2026-11-03 16:19:28.180274335 +0000 UTC m=+311.873545356" Nov 03 16:19:28.321638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:28.321600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:29.201186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.201154 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj"] Nov 03 16:19:29.204361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.204344 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.231525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.231499 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-s7mlz\"" Nov 03 16:19:29.231657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.231574 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 16:19:29.231657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.231586 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:19:29.233398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.233374 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj"] Nov 03 16:19:29.298926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.298901 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fccacb21-900e-46f6-8b24-24800959c566-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj\" (UID: \"fccacb21-900e-46f6-8b24-24800959c566\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.299082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.298939 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd6b\" (UniqueName: \"kubernetes.io/projected/fccacb21-900e-46f6-8b24-24800959c566-kube-api-access-9fd6b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj\" (UID: \"fccacb21-900e-46f6-8b24-24800959c566\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.301454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.301412 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc"] Nov 03 16:19:29.304612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.304598 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:29.320871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.320855 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bfl7l\"" Nov 03 16:19:29.321361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.321346 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 16:19:29.331501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.331481 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc"] Nov 03 16:19:29.399950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.399918 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fccacb21-900e-46f6-8b24-24800959c566-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj\" (UID: \"fccacb21-900e-46f6-8b24-24800959c566\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.400096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.399958 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd6b\" (UniqueName: \"kubernetes.io/projected/fccacb21-900e-46f6-8b24-24800959c566-kube-api-access-9fd6b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj\" (UID: \"fccacb21-900e-46f6-8b24-24800959c566\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.400096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.399987 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76gx\" (UniqueName: \"kubernetes.io/projected/9f80cd1d-c291-40d4-866b-6b1ee9005dac-kube-api-access-w76gx\") pod \"dns-operator-controller-manager-8557cbdb84-jz5xc\" (UID: \"9f80cd1d-c291-40d4-866b-6b1ee9005dac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:29.400302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.400283 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fccacb21-900e-46f6-8b24-24800959c566-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj\" (UID: \"fccacb21-900e-46f6-8b24-24800959c566\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.424912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.424880 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd6b\" (UniqueName: \"kubernetes.io/projected/fccacb21-900e-46f6-8b24-24800959c566-kube-api-access-9fd6b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj\" (UID: \"fccacb21-900e-46f6-8b24-24800959c566\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.500762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.500676 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76gx\" (UniqueName: \"kubernetes.io/projected/9f80cd1d-c291-40d4-866b-6b1ee9005dac-kube-api-access-w76gx\") pod \"dns-operator-controller-manager-8557cbdb84-jz5xc\" (UID: \"9f80cd1d-c291-40d4-866b-6b1ee9005dac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:29.515598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.515571 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" Nov 03 16:19:29.536700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.536664 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76gx\" (UniqueName: \"kubernetes.io/projected/9f80cd1d-c291-40d4-866b-6b1ee9005dac-kube-api-access-w76gx\") pod \"dns-operator-controller-manager-8557cbdb84-jz5xc\" (UID: \"9f80cd1d-c291-40d4-866b-6b1ee9005dac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:29.616135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.616099 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:29.839884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.839851 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj"] Nov 03 16:19:29.843520 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:29.843486 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccacb21_900e_46f6_8b24_24800959c566.slice/crio-608652592b4aa292050fc80748a5d6661f070bc6bdb51d48b5985948a433ca5d WatchSource:0}: Error finding container 608652592b4aa292050fc80748a5d6661f070bc6bdb51d48b5985948a433ca5d: Status 404 returned error can't find the container with id 608652592b4aa292050fc80748a5d6661f070bc6bdb51d48b5985948a433ca5d Nov 03 16:19:29.919797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:29.919772 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc"] Nov 03 16:19:29.922116 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:29.922092 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f80cd1d_c291_40d4_866b_6b1ee9005dac.slice/crio-fd9c4743bec6e63631349bf89f928480f9d40caf9b48b759579c8f5ec3eb18ac WatchSource:0}: Error finding container fd9c4743bec6e63631349bf89f928480f9d40caf9b48b759579c8f5ec3eb18ac: Status 404 returned error can't find the container with id fd9c4743bec6e63631349bf89f928480f9d40caf9b48b759579c8f5ec3eb18ac Nov 03 16:19:30.046408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:30.046366 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:30.098340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:30.098240 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" event={"ID":"9f80cd1d-c291-40d4-866b-6b1ee9005dac","Type":"ContainerStarted","Data":"fd9c4743bec6e63631349bf89f928480f9d40caf9b48b759579c8f5ec3eb18ac"} Nov 03 16:19:30.099702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:30.099681 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" event={"ID":"fccacb21-900e-46f6-8b24-24800959c566","Type":"ContainerStarted","Data":"608652592b4aa292050fc80748a5d6661f070bc6bdb51d48b5985948a433ca5d"} Nov 03 16:19:30.239996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:30.239964 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:31.200445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:31.200383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:31.606855 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:31.606823 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:31.792589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:31.792544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:31.973962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:31.973931 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:32.113804 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:32.113769 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" event={"ID":"9f80cd1d-c291-40d4-866b-6b1ee9005dac","Type":"ContainerStarted","Data":"b4ab5b51665db1233998832dda98466bd784ce3d668d2ba58e76fa6bdcab40f2"} Nov 03 16:19:32.114085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:32.113903 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:32.120754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:32.120677 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" event={"ID":"fccacb21-900e-46f6-8b24-24800959c566","Type":"ContainerStarted","Data":"b95aea85740c9593296173992485a3ceaecd21b44ca8f5682827d0bdb21f4d42"} Nov 03 16:19:32.187760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:32.187703 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" podStartSLOduration=1.242217207 podStartE2EDuration="3.187687783s" totalImagesPullingTime="1.945470576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:32.18664964 +0000 UTC m=+315.879920663" watchObservedRunningTime="2026-11-03 16:19:32.187687783 +0000 UTC m=+315.880958807" Nov 03 16:19:32.245083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:32.245035 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lxmhj" podStartSLOduration=1.8035518640000001 podStartE2EDuration="3.245021693s" totalImagesPullingTime="1.441469829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:32.243674198 +0000 UTC m=+315.936945220" watchObservedRunningTime="2026-11-03 16:19:32.245021693 +0000 UTC m=+315.938292714" Nov 03 16:19:32.914668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:32.914630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:33.770739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:33.770704 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:34.053012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:34.052979 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:34.454385 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:34.454295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:35.074651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:35.074624 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5ldg2" Nov 03 16:19:35.908612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:35.908571 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:36.449688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:36.449658 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:37.003681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.003650 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb"] Nov 03 16:19:37.007325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.007306 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.027267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.027243 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vfdxp\"" Nov 03 16:19:37.039818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.039793 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb"] Nov 03 16:19:37.065193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.065162 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9qr\" (UniqueName: \"kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr\") pod \"kuadrant-operator-controller-manager-678d87766f-jtdbb\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.065322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.065227 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jtdbb\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.165764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.165729 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9qr\" (UniqueName: \"kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr\") pod \"kuadrant-operator-controller-manager-678d87766f-jtdbb\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.165940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.165785 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jtdbb\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.166126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.166110 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jtdbb\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.191365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.191336 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9qr\" (UniqueName: \"kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr\") pod \"kuadrant-operator-controller-manager-678d87766f-jtdbb\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.319709 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.319677 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:37.651207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:37.651136 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb"] Nov 03 16:19:37.664805 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:37.664778 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e6ae4a_0f66_47e2_bd89_af9390b90862.slice/crio-ddaa4df86b6d21354b9307b2694a375c989bdcb77e66ad6b0e2aba4681d94b4d WatchSource:0}: Error finding container ddaa4df86b6d21354b9307b2694a375c989bdcb77e66ad6b0e2aba4681d94b4d: Status 404 returned error can't find the container with id ddaa4df86b6d21354b9307b2694a375c989bdcb77e66ad6b0e2aba4681d94b4d Nov 03 16:19:38.155748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:38.155716 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" event={"ID":"92e6ae4a-0f66-47e2-bd89-af9390b90862","Type":"ContainerStarted","Data":"ddaa4df86b6d21354b9307b2694a375c989bdcb77e66ad6b0e2aba4681d94b4d"} Nov 03 16:19:39.095166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:39.095126 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mqxzs" Nov 03 16:19:39.415567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:39.415493 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:39.566556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:39.566521 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:41.621599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:41.621439 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:41.813167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:41.813135 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:42.014834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:42.014742 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:42.186574 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:42.186541 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" event={"ID":"92e6ae4a-0f66-47e2-bd89-af9390b90862","Type":"ContainerStarted","Data":"47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345"} Nov 03 16:19:42.186794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:42.186683 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:42.263848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:42.263793 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" podStartSLOduration=2.270408388 podStartE2EDuration="6.263778977s" totalImagesPullingTime="3.993370589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:42.262914795 +0000 UTC m=+325.956185830" watchObservedRunningTime="2026-11-03 16:19:42.263778977 +0000 UTC m=+325.957049999" Nov 03 16:19:42.314734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:42.314700 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:42.902638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:42.902608 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:43.127375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.127345 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jz5xc" Nov 03 16:19:43.162939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.162855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:43.178813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.178785 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr"] Nov 03 16:19:43.182236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.182220 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.212464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.212418 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qxrjm\"" Nov 03 16:19:43.234033 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.233992 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr"] Nov 03 16:19:43.319294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319262 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319307 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319456 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319499 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319525 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319564 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319636 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq2c\" (UniqueName: \"kubernetes.io/projected/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-kube-api-access-9rq2c\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319682 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.319805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.319710 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.408823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.408789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:43.420878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420804 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.420878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420843 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.420878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420868 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420887 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq2c\" (UniqueName: \"kubernetes.io/projected/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-kube-api-access-9rq2c\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420918 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420937 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.420975 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.421017 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.421067 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.421268 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.421369 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.421517 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.421826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.421801 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.422266 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.422240 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.423555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.423529 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.423755 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.423733 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.450212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.450171 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq2c\" (UniqueName: \"kubernetes.io/projected/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-kube-api-access-9rq2c\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.453719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.453686 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-swdzr\" (UID: \"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.497191 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.497153 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:43.617237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.617179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:43.846034 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.846010 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr"] Nov 03 16:19:43.848382 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:43.848350 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d72e5f_33ca_45e2_b4dd_bd70d15ab4e0.slice/crio-97b57636f97e6ccccc999cbc20b71544eedefff2f57c736511a12029bab93e52 WatchSource:0}: Error finding container 97b57636f97e6ccccc999cbc20b71544eedefff2f57c736511a12029bab93e52: Status 404 returned error can't find the container with id 97b57636f97e6ccccc999cbc20b71544eedefff2f57c736511a12029bab93e52 Nov 03 16:19:43.850496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.850466 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:19:43.850576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.850534 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:19:43.850576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:43.850563 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:19:44.203835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.203733 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" event={"ID":"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0","Type":"ContainerStarted","Data":"28b4a039cdf8d232577b246eec08171eaa22d2fd33d54cba4c705be59d10d980"} Nov 03 16:19:44.203835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.203787 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" event={"ID":"c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0","Type":"ContainerStarted","Data":"97b57636f97e6ccccc999cbc20b71544eedefff2f57c736511a12029bab93e52"} Nov 03 16:19:44.281132 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.281066 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" podStartSLOduration=1.2810500839999999 podStartE2EDuration="1.281050084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:44.279281855 +0000 UTC m=+327.972552878" watchObservedRunningTime="2026-11-03 16:19:44.281050084 +0000 UTC m=+327.974321111" Nov 03 16:19:44.497551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.497463 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:44.502271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.502246 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:44.540908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.540878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:44.795332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:44.795297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:45.208535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:45.208449 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:45.209535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:45.209515 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-swdzr" Nov 03 16:19:45.584749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:45.584719 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:45.744407 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:45.744376 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:46.165644 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:46.165616 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:46.391867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:46.391833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:47.683415 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.683379 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7"] Nov 03 16:19:47.687028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.687011 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.707895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.707868 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 16:19:47.718093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.718068 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 16:19:47.718238 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.718129 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-48w2h\"" Nov 03 16:19:47.739647 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.739613 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 16:19:47.741680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.741621 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7"] Nov 03 16:19:47.758360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.758326 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22bcb2db-3d3f-49cb-b196-de4ca60476ef-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.758636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.758406 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22bcb2db-3d3f-49cb-b196-de4ca60476ef-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.758636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.758554 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wkqp\" (UniqueName: \"kubernetes.io/projected/22bcb2db-3d3f-49cb-b196-de4ca60476ef-kube-api-access-9wkqp\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.758636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.758589 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22bcb2db-3d3f-49cb-b196-de4ca60476ef-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.859139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.859094 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22bcb2db-3d3f-49cb-b196-de4ca60476ef-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.859359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.859165 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkqp\" (UniqueName: \"kubernetes.io/projected/22bcb2db-3d3f-49cb-b196-de4ca60476ef-kube-api-access-9wkqp\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.859359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.859197 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22bcb2db-3d3f-49cb-b196-de4ca60476ef-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.859359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.859221 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22bcb2db-3d3f-49cb-b196-de4ca60476ef-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.859720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.859698 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22bcb2db-3d3f-49cb-b196-de4ca60476ef-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.859791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.859723 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22bcb2db-3d3f-49cb-b196-de4ca60476ef-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.860100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.860077 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22bcb2db-3d3f-49cb-b196-de4ca60476ef-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:47.881675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:47.881640 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkqp\" (UniqueName: \"kubernetes.io/projected/22bcb2db-3d3f-49cb-b196-de4ca60476ef-kube-api-access-9wkqp\") pod \"kserve-module-controller-manager-7566cf5fc8-kswm7\" (UID: \"22bcb2db-3d3f-49cb-b196-de4ca60476ef\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:48.000749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:48.000652 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:48.319497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:48.319463 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7"] Nov 03 16:19:48.322362 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:19:48.322327 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bcb2db_3d3f_49cb_b196_de4ca60476ef.slice/crio-927c0e2c4676b2fdfa08367a4300b08816cc759f97df8b462e5e2a4a26521ca4 WatchSource:0}: Error finding container 927c0e2c4676b2fdfa08367a4300b08816cc759f97df8b462e5e2a4a26521ca4: Status 404 returned error can't find the container with id 927c0e2c4676b2fdfa08367a4300b08816cc759f97df8b462e5e2a4a26521ca4 Nov 03 16:19:49.228916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:49.228874 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" event={"ID":"22bcb2db-3d3f-49cb-b196-de4ca60476ef","Type":"ContainerStarted","Data":"927c0e2c4676b2fdfa08367a4300b08816cc759f97df8b462e5e2a4a26521ca4"} Nov 03 16:19:50.381578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:50.381532 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:50.541082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:50.541046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:51.240109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:51.240077 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" event={"ID":"22bcb2db-3d3f-49cb-b196-de4ca60476ef","Type":"ContainerStarted","Data":"8e2efa8666e798aa53bd954d27ab661ad2a5b7c2fe5c6233a1167ea42233c19c"} Nov 03 16:19:51.240293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:51.240196 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:19:51.295962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:51.295903 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" podStartSLOduration=1.991622177 podStartE2EDuration="4.295887357s" totalImagesPullingTime="2.30426518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:51.29400755 +0000 UTC m=+334.987278572" watchObservedRunningTime="2026-11-03 16:19:51.295887357 +0000 UTC m=+334.989158379" Nov 03 16:19:53.200047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:53.200011 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:19:56.155918 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:19:56.155881 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:19:56.249347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:56.249318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:56.418089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:56.418004 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:19:57.248794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:19:57.248761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:03.005136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:03.005105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:03.178935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:03.178903 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:06.521478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:06.521412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:06.678067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:06.678029 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:06.921192 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:06.921159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:07.150852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:07.150815 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:07.835811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:07.835775 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:07.993058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:07.993030 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:09.909840 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:09.909807 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:10.070969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:10.070932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:12.246352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:12.246323 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kswm7" Nov 03 16:20:15.318210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:15.318171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:15.459615 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:15.459583 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:18.059692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:18.059657 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:26.155745 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:20:26.155704 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:20:44.586708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.586670 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:44.738751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.738716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:44.858210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.858115 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b6b4p"] Nov 03 16:20:44.865321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.865302 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:44.889289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.888629 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 16:20:44.889289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.888771 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-n7jsp\"" Nov 03 16:20:44.899918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.899884 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b6b4p"] Nov 03 16:20:44.900089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.899994 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72"] Nov 03 16:20:44.904859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.904826 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:44.927151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.927106 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72"] Nov 03 16:20:44.934190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.934162 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:44.938062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.938041 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dmg72\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:44.938140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.938074 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6lj\" (UniqueName: \"kubernetes.io/projected/ff6e4b8d-1c33-4e59-9aad-512e10134470-kube-api-access-8k6lj\") pod \"kserve-controller-manager-776cc66d47-b6b4p\" (UID: \"ff6e4b8d-1c33-4e59-9aad-512e10134470\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:44.938194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.938179 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rls5l\" (UniqueName: \"kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l\") pod \"llmisvc-controller-manager-869d5dcc5d-dmg72\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:44.938239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.938203 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff6e4b8d-1c33-4e59-9aad-512e10134470-cert\") pod \"kserve-controller-manager-776cc66d47-b6b4p\" (UID: \"ff6e4b8d-1c33-4e59-9aad-512e10134470\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:44.943232 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.943201 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5lqff"] Nov 03 16:20:44.947337 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.947317 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:44.971019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.970995 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5lqff"] Nov 03 16:20:44.984020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.983983 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2lkcv\"" Nov 03 16:20:44.984020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.983987 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-sj9m8\"" Nov 03 16:20:44.984204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.984031 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 16:20:44.984204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.984080 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 16:20:44.999199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:44.999172 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-625qt"] Nov 03 16:20:45.002711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.002687 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.026035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.025987 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mkxlc\"" Nov 03 16:20:45.026195 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.026135 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 16:20:45.026265 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.026253 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 16:20:45.028485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.028444 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-625qt"] Nov 03 16:20:45.039595 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039564 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-metrics-certs\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.039719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039607 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tb6\" (UniqueName: \"kubernetes.io/projected/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-kube-api-access-44tb6\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.039762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039729 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rls5l\" (UniqueName: \"kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l\") pod \"llmisvc-controller-manager-869d5dcc5d-dmg72\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:45.039807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039758 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff6e4b8d-1c33-4e59-9aad-512e10134470-cert\") pod \"kserve-controller-manager-776cc66d47-b6b4p\" (UID: \"ff6e4b8d-1c33-4e59-9aad-512e10134470\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:45.039807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039786 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlvn\" (UniqueName: \"kubernetes.io/projected/8f60d60e-9283-4377-b974-fe3e97be1bff-kube-api-access-wjlvn\") pod \"model-serving-api-75bd89fdfc-5lqff\" (UID: \"8f60d60e-9283-4377-b974-fe3e97be1bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.039902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039811 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f60d60e-9283-4377-b974-fe3e97be1bff-tls-certs\") pod \"model-serving-api-75bd89fdfc-5lqff\" (UID: \"8f60d60e-9283-4377-b974-fe3e97be1bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.039902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039850 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dmg72\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:45.039902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039873 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6lj\" (UniqueName: \"kubernetes.io/projected/ff6e4b8d-1c33-4e59-9aad-512e10134470-kube-api-access-8k6lj\") pod \"kserve-controller-manager-776cc66d47-b6b4p\" (UID: \"ff6e4b8d-1c33-4e59-9aad-512e10134470\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:45.040047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.039981 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-cert\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.042496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.042476 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dmg72\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:45.042601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.042474 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff6e4b8d-1c33-4e59-9aad-512e10134470-cert\") pod \"kserve-controller-manager-776cc66d47-b6b4p\" (UID: \"ff6e4b8d-1c33-4e59-9aad-512e10134470\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:45.063154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.063122 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rls5l\" (UniqueName: \"kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l\") pod \"llmisvc-controller-manager-869d5dcc5d-dmg72\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:45.066500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.066480 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6lj\" (UniqueName: \"kubernetes.io/projected/ff6e4b8d-1c33-4e59-9aad-512e10134470-kube-api-access-8k6lj\") pod \"kserve-controller-manager-776cc66d47-b6b4p\" (UID: \"ff6e4b8d-1c33-4e59-9aad-512e10134470\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:45.099288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.099266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:45.140529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.140458 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlvn\" (UniqueName: \"kubernetes.io/projected/8f60d60e-9283-4377-b974-fe3e97be1bff-kube-api-access-wjlvn\") pod \"model-serving-api-75bd89fdfc-5lqff\" (UID: \"8f60d60e-9283-4377-b974-fe3e97be1bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.140529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.140492 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f60d60e-9283-4377-b974-fe3e97be1bff-tls-certs\") pod \"model-serving-api-75bd89fdfc-5lqff\" (UID: \"8f60d60e-9283-4377-b974-fe3e97be1bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.140727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.140536 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-cert\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.140727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.140561 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-metrics-certs\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.140727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.140580 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tb6\" (UniqueName: \"kubernetes.io/projected/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-kube-api-access-44tb6\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.143147 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.143109 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-cert\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.143249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.143135 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f60d60e-9283-4377-b974-fe3e97be1bff-tls-certs\") pod \"model-serving-api-75bd89fdfc-5lqff\" (UID: \"8f60d60e-9283-4377-b974-fe3e97be1bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.143249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.143135 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-metrics-certs\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.161817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.161797 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tb6\" (UniqueName: \"kubernetes.io/projected/e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd-kube-api-access-44tb6\") pod \"odh-model-controller-769875db56-625qt\" (UID: \"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd\") " pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.161817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.161815 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlvn\" (UniqueName: \"kubernetes.io/projected/8f60d60e-9283-4377-b974-fe3e97be1bff-kube-api-access-wjlvn\") pod \"model-serving-api-75bd89fdfc-5lqff\" (UID: \"8f60d60e-9283-4377-b974-fe3e97be1bff\") " pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.177688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.177664 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:45.218017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.217919 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:45.260204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.260177 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:45.319186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.319150 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:45.430098 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.430027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:45.475769 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.475748 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b6b4p"] Nov 03 16:20:45.477255 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:20:45.477222 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6e4b8d_1c33_4e59_9aad_512e10134470.slice/crio-02212b33e78f0be82654889407f3bb76bd797922a521c5fede81c3f46670833c WatchSource:0}: Error finding container 02212b33e78f0be82654889407f3bb76bd797922a521c5fede81c3f46670833c: Status 404 returned error can't find the container with id 02212b33e78f0be82654889407f3bb76bd797922a521c5fede81c3f46670833c Nov 03 16:20:45.496259 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.496232 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" event={"ID":"ff6e4b8d-1c33-4e59-9aad-512e10134470","Type":"ContainerStarted","Data":"02212b33e78f0be82654889407f3bb76bd797922a521c5fede81c3f46670833c"} Nov 03 16:20:45.562109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.562071 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72"] Nov 03 16:20:45.564662 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:20:45.564633 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb7e98e62_20a2_462f_a268_342dd93bf569.slice/crio-50f102ca5ddb5a9dc6c1fdd7e17795a49a6cc79abcad1006fa07c6b66dcdfb93 WatchSource:0}: Error finding container 50f102ca5ddb5a9dc6c1fdd7e17795a49a6cc79abcad1006fa07c6b66dcdfb93: Status 404 returned error can't find the container with id 50f102ca5ddb5a9dc6c1fdd7e17795a49a6cc79abcad1006fa07c6b66dcdfb93 Nov 03 16:20:45.592682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.592662 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5lqff"] Nov 03 16:20:45.594807 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:20:45.594782 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f60d60e_9283_4377_b974_fe3e97be1bff.slice/crio-40223dbd712574b166eda55d02f6cc79427d15dc2756ad62e4a94595e8068246 WatchSource:0}: Error finding container 40223dbd712574b166eda55d02f6cc79427d15dc2756ad62e4a94595e8068246: Status 404 returned error can't find the container with id 40223dbd712574b166eda55d02f6cc79427d15dc2756ad62e4a94595e8068246 Nov 03 16:20:45.621756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:45.621733 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-625qt"] Nov 03 16:20:45.623333 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:20:45.623310 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f1bf7d_8cc6_40ff_8d61_4f4940f80afd.slice/crio-137930dee5332cda1d240919bb7228ddff51fe9cc11ef4b0ffe0c0367d359b20 WatchSource:0}: Error finding container 137930dee5332cda1d240919bb7228ddff51fe9cc11ef4b0ffe0c0367d359b20: Status 404 returned error can't find the container with id 137930dee5332cda1d240919bb7228ddff51fe9cc11ef4b0ffe0c0367d359b20 Nov 03 16:20:46.504917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:46.504870 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" event={"ID":"b7e98e62-20a2-462f-a268-342dd93bf569","Type":"ContainerStarted","Data":"50f102ca5ddb5a9dc6c1fdd7e17795a49a6cc79abcad1006fa07c6b66dcdfb93"} Nov 03 16:20:46.507126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:46.507099 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" event={"ID":"8f60d60e-9283-4377-b974-fe3e97be1bff","Type":"ContainerStarted","Data":"40223dbd712574b166eda55d02f6cc79427d15dc2756ad62e4a94595e8068246"} Nov 03 16:20:46.508914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:46.508892 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-625qt" event={"ID":"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd","Type":"ContainerStarted","Data":"137930dee5332cda1d240919bb7228ddff51fe9cc11ef4b0ffe0c0367d359b20"} Nov 03 16:20:47.564621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:47.564583 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:47.719382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:47.719345 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:47.885485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:47.885387 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:48.044617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:48.044529 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:48.534827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:48.534792 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" event={"ID":"8f60d60e-9283-4377-b974-fe3e97be1bff","Type":"ContainerStarted","Data":"976169c6ab6a6a005c3a79738fa9c96773c4eb422c648e8a6515408c0327e372"} Nov 03 16:20:48.534979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:48.534959 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:20:48.604221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:48.604156 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" podStartSLOduration=2.124274964 podStartE2EDuration="4.604135528s" totalImagesPullingTime="2.479860564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:48.601798145 +0000 UTC m=+392.295069167" watchObservedRunningTime="2026-11-03 16:20:48.604135528 +0000 UTC m=+392.297406552" Nov 03 16:20:48.683685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:48.683658 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:48.830851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:48.830814 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:20:49.542844 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.542804 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-625qt" event={"ID":"e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd","Type":"ContainerStarted","Data":"008b1e44d9b0d282f885283e159b23863f906a2f597aa60696e8ca386e531f46"} Nov 03 16:20:49.543106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.543083 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:20:49.545148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.545097 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" event={"ID":"b7e98e62-20a2-462f-a268-342dd93bf569","Type":"ContainerStarted","Data":"91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa"} Nov 03 16:20:49.545339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.545316 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:20:49.546658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.546639 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" event={"ID":"ff6e4b8d-1c33-4e59-9aad-512e10134470","Type":"ContainerStarted","Data":"bdbe35dc562e9740557c6a5d515424c301deaaf32cbbecbb54f1d02c5e091a05"} Nov 03 16:20:49.546741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.546725 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:20:49.588100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.588051 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-625qt" podStartSLOduration=2.105139536 podStartE2EDuration="5.588037895s" totalImagesPullingTime="3.482898359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:49.586014639 +0000 UTC m=+393.279285662" watchObservedRunningTime="2026-11-03 16:20:49.588037895 +0000 UTC m=+393.281308916" Nov 03 16:20:49.629877 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.629826 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" podStartSLOduration=2.5288501070000002 podStartE2EDuration="5.629811716s" totalImagesPullingTime="3.100961609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:49.628134481 +0000 UTC m=+393.321405503" watchObservedRunningTime="2026-11-03 16:20:49.629811716 +0000 UTC m=+393.323082763" Nov 03 16:20:49.677259 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:49.677210 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" podStartSLOduration=2.686923715 podStartE2EDuration="5.677193074s" totalImagesPullingTime="2.990269359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:49.676003091 +0000 UTC m=+393.369274146" watchObservedRunningTime="2026-11-03 16:20:49.677193074 +0000 UTC m=+393.370464100" Nov 03 16:20:56.156413 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:20:56.156377 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:20:59.550586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:20:59.550510 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-5lqff" Nov 03 16:21:00.554187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:21:00.554155 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-625qt" Nov 03 16:21:07.203177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:21:07.203131 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:20.553687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:21:20.553658 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:21:20.556684 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:21:20.556665 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b6b4p" Nov 03 16:21:26.156371 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:21:26.156331 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:21:36.225522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:21:36.225493 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:36.402360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:21:36.402330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:21:56.155207 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:21:56.155162 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:22:07.426803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:22:07.426767 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:22:26.155280 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:22:26.155243 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:22:56.154921 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:22:56.154885 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:23:07.431109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:23:07.431076 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:26.155372 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:23:26.155338 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:23:33.661133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:23:33.661099 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:33.801952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:23:33.801919 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:23:56.156260 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:23:56.156215 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:24:07.545210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:24:07.545124 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:24:16.507511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:24:16.507482 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:24:16.508128 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:24:16.508108 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:24:26.156349 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:24:26.156311 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:24:56.155734 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:24:56.155693 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:25:02.625460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:25:02.625402 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:07.677210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:25:07.677166 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:18.247540 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:25:18.247508 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:26.156551 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:25:26.156516 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:25:30.439714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:25:30.439614 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:25:56.155601 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:25:56.155564 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:26:07.793215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:26:07.793176 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:10.924140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:26:10.924103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:12.263587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:26:12.263529 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:26.155882 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:26:26.155844 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:26:55.801372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:26:55.801340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:55.969814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:26:55.969782 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:26:56.155395 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:26:56.155318 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:27:07.917715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:27:07.917642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:21.329481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:27:21.329446 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:26.155530 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:27:26.155491 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:27:32.223300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:27:32.223266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:27:56.156188 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:27:56.156152 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:28:08.057579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:28:08.057542 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:28:26.155652 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:28:26.155614 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:28:56.154945 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:28:56.154908 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:29:08.139782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:29:08.139731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:29:16.545264 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:29:16.545236 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:29:16.547449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:29:16.547406 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:29:26.155031 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:29:26.154991 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:29:56.155976 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:29:56.155937 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:30:02.313978 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.313897 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jtdbb_92e6ae4a-0f66-47e2-bd89-af9390b90862/manager/0.log" Nov 03 16:30:02.552237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.552202 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb"] Nov 03 16:30:02.552530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.552505 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" podUID="92e6ae4a-0f66-47e2-bd89-af9390b90862" containerName="manager" containerID="cri-o://47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345" gracePeriod=10 Nov 03 16:30:02.578134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.578109 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6"] Nov 03 16:30:02.581473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.581455 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.617442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.617399 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtqt6\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.617598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.617536 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99c5p\" (UniqueName: \"kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p\") pod \"kuadrant-operator-controller-manager-678d87766f-wtqt6\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.618031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.618012 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6"] Nov 03 16:30:02.718086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.718052 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99c5p\" (UniqueName: \"kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p\") pod \"kuadrant-operator-controller-manager-678d87766f-wtqt6\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.718262 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.718128 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtqt6\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.718553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.718530 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtqt6\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.744238 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.744207 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99c5p\" (UniqueName: \"kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p\") pod \"kuadrant-operator-controller-manager-678d87766f-wtqt6\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:02.811756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.811727 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:30:02.920248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.920152 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr\" (UniqueName: \"kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr\") pod \"92e6ae4a-0f66-47e2-bd89-af9390b90862\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " Nov 03 16:30:02.920248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.920227 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume\") pod \"92e6ae4a-0f66-47e2-bd89-af9390b90862\" (UID: \"92e6ae4a-0f66-47e2-bd89-af9390b90862\") " Nov 03 16:30:02.920615 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.920586 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume" pod "92e6ae4a-0f66-47e2-bd89-af9390b90862" (UID: "92e6ae4a-0f66-47e2-bd89-af9390b90862"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:30:02.922555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.922531 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr" pod "92e6ae4a-0f66-47e2-bd89-af9390b90862" (UID: "92e6ae4a-0f66-47e2-bd89-af9390b90862"). InnerVolumeSpecName "kube-api-access-qx9qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:02.932641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:02.932618 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:03.020997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.020962 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9qr\" (UniqueName: \"kubernetes.io/projected/92e6ae4a-0f66-47e2-bd89-af9390b90862-kube-api-access-qx9qr\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:30:03.020997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.020994 4539 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/92e6ae4a-0f66-47e2-bd89-af9390b90862-extensions-socket-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:30:03.249652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.249626 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6"] Nov 03 16:30:03.252397 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:30:03.252372 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a057da_36a9_417d_a66d_3f56cc6c6fff.slice/crio-a183e32c11f668ad90caec9e1bc66e33bf8aee5dffa03186d5ab1f9760a65a7e WatchSource:0}: Error finding container a183e32c11f668ad90caec9e1bc66e33bf8aee5dffa03186d5ab1f9760a65a7e: Status 404 returned error can't find the container with id a183e32c11f668ad90caec9e1bc66e33bf8aee5dffa03186d5ab1f9760a65a7e Nov 03 16:30:03.300360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.300327 4539 generic.go:356] "Generic (PLEG): container finished" podID="92e6ae4a-0f66-47e2-bd89-af9390b90862" containerID="47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345" exitCode=0 Nov 03 16:30:03.300500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.300439 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" Nov 03 16:30:03.300500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.300449 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" event={"ID":"92e6ae4a-0f66-47e2-bd89-af9390b90862","Type":"ContainerDied","Data":"47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345"} Nov 03 16:30:03.300626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.300506 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb" event={"ID":"92e6ae4a-0f66-47e2-bd89-af9390b90862","Type":"ContainerDied","Data":"ddaa4df86b6d21354b9307b2694a375c989bdcb77e66ad6b0e2aba4681d94b4d"} Nov 03 16:30:03.300626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.300537 4539 scope.go:122] "RemoveContainer" containerID="47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345" Nov 03 16:30:03.303731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.303700 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" event={"ID":"a5a057da-36a9-417d-a66d-3f56cc6c6fff","Type":"ContainerStarted","Data":"a183e32c11f668ad90caec9e1bc66e33bf8aee5dffa03186d5ab1f9760a65a7e"} Nov 03 16:30:03.310093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.310074 4539 scope.go:122] "RemoveContainer" containerID="47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345" Nov 03 16:30:03.310373 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:30:03.310352 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345\": container with ID starting with 47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345 not found: ID does not exist" containerID="47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345" Nov 03 16:30:03.310448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.310383 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345"} err="failed to get container status \"47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345\": rpc error: code = NotFound desc = could not find container \"47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345\": container with ID starting with 47f8b8527a5f6eee1f4e792da2229e576e034cc4032dbb707396ee7f8912b345 not found: ID does not exist" Nov 03 16:30:03.355751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.355713 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb"] Nov 03 16:30:03.360019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:03.359993 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtdbb"] Nov 03 16:30:04.311562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:04.311530 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" event={"ID":"a5a057da-36a9-417d-a66d-3f56cc6c6fff","Type":"ContainerStarted","Data":"b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b"} Nov 03 16:30:04.311764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:04.311656 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:04.375009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:04.374949 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" podStartSLOduration=2.374922908 podStartE2EDuration="2.374922908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:04.36313731 +0000 UTC m=+948.056408332" watchObservedRunningTime="2026-11-03 16:30:04.374922908 +0000 UTC m=+948.068193932" Nov 03 16:30:04.467073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:04.467040 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e6ae4a-0f66-47e2-bd89-af9390b90862" path="/var/lib/kubelet/pods/92e6ae4a-0f66-47e2-bd89-af9390b90862/volumes" Nov 03 16:30:08.304008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:08.303973 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:15.319562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:15.319535 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 16:30:21.361394 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:21.361353 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:21.510636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:21.510589 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:21.658321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:21.658236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:22.208094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.208056 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j982c"] Nov 03 16:30:22.208483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.208468 4539 cpu_manager.go:414] "containerMap: removing container" podUID="92e6ae4a-0f66-47e2-bd89-af9390b90862" containerName="manager" Nov 03 16:30:22.208549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.208485 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e6ae4a-0f66-47e2-bd89-af9390b90862" containerName="manager" Nov 03 16:30:22.208591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.208580 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e6ae4a-0f66-47e2-bd89-af9390b90862" containerName="manager" Nov 03 16:30:22.212634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.212618 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:22.229736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.229712 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j982c"] Nov 03 16:30:22.237559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.237539 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvhsq\"" Nov 03 16:30:22.290553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.290521 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4j5\" (UniqueName: \"kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5\") pod \"authorino-57f7f8659f-j982c\" (UID: \"896bbdd8-02b1-47cc-9bdc-191c348be1e1\") " pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:22.330061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.330031 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:22.391775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.391747 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4j5\" (UniqueName: \"kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5\") pod \"authorino-57f7f8659f-j982c\" (UID: \"896bbdd8-02b1-47cc-9bdc-191c348be1e1\") " pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:22.416148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.416109 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4j5\" (UniqueName: \"kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5\") pod \"authorino-57f7f8659f-j982c\" (UID: \"896bbdd8-02b1-47cc-9bdc-191c348be1e1\") " pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:22.471461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.471368 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:22.534873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.534837 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:22.835299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.835211 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j982c"] Nov 03 16:30:22.837636 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:30:22.837606 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod896bbdd8_02b1_47cc_9bdc_191c348be1e1.slice/crio-a68cec3a57a6d71a8e8bcab205aeef7c76657eaef3372c907a34c5f28daf2d37 WatchSource:0}: Error finding container a68cec3a57a6d71a8e8bcab205aeef7c76657eaef3372c907a34c5f28daf2d37: Status 404 returned error can't find the container with id a68cec3a57a6d71a8e8bcab205aeef7c76657eaef3372c907a34c5f28daf2d37 Nov 03 16:30:22.839113 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:22.839092 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:30:23.406100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:23.406064 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j982c" event={"ID":"896bbdd8-02b1-47cc-9bdc-191c348be1e1","Type":"ContainerStarted","Data":"a68cec3a57a6d71a8e8bcab205aeef7c76657eaef3372c907a34c5f28daf2d37"} Nov 03 16:30:25.419781 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:25.419747 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j982c" event={"ID":"896bbdd8-02b1-47cc-9bdc-191c348be1e1","Type":"ContainerStarted","Data":"a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7"} Nov 03 16:30:25.457289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:25.457224 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-j982c" podStartSLOduration=1.820841928 podStartE2EDuration="3.457207271s" totalImagesPullingTime="1.636365343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:25.455813988 +0000 UTC m=+969.149085009" watchObservedRunningTime="2026-11-03 16:30:25.457207271 +0000 UTC m=+969.150478703" Nov 03 16:30:26.155377 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:30:26.155345 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:30:26.903887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:26.903849 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j982c"] Nov 03 16:30:27.434352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:27.434289 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-j982c" podUID="896bbdd8-02b1-47cc-9bdc-191c348be1e1" containerName="authorino" containerID="cri-o://a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7" gracePeriod=30 Nov 03 16:30:27.687113 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:27.687057 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:27.843371 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:27.843344 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5\" (UniqueName: \"kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5\") pod \"896bbdd8-02b1-47cc-9bdc-191c348be1e1\" (UID: \"896bbdd8-02b1-47cc-9bdc-191c348be1e1\") " Nov 03 16:30:27.845516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:27.845495 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5" pod "896bbdd8-02b1-47cc-9bdc-191c348be1e1" (UID: "896bbdd8-02b1-47cc-9bdc-191c348be1e1"). InnerVolumeSpecName "kube-api-access-9r4j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:27.944845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:27.944748 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4j5\" (UniqueName: \"kubernetes.io/projected/896bbdd8-02b1-47cc-9bdc-191c348be1e1-kube-api-access-9r4j5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:30:28.440369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.440332 4539 generic.go:356] "Generic (PLEG): container finished" podID="896bbdd8-02b1-47cc-9bdc-191c348be1e1" containerID="a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7" exitCode=0 Nov 03 16:30:28.440552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.440379 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j982c" Nov 03 16:30:28.440552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.440389 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j982c" event={"ID":"896bbdd8-02b1-47cc-9bdc-191c348be1e1","Type":"ContainerDied","Data":"a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7"} Nov 03 16:30:28.440552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.440447 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j982c" event={"ID":"896bbdd8-02b1-47cc-9bdc-191c348be1e1","Type":"ContainerDied","Data":"a68cec3a57a6d71a8e8bcab205aeef7c76657eaef3372c907a34c5f28daf2d37"} Nov 03 16:30:28.440552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.440474 4539 scope.go:122] "RemoveContainer" containerID="a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7" Nov 03 16:30:28.449334 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.449313 4539 scope.go:122] "RemoveContainer" containerID="a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7" Nov 03 16:30:28.449637 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:30:28.449612 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7\": container with ID starting with a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7 not found: ID does not exist" containerID="a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7" Nov 03 16:30:28.449725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.449652 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7"} err="failed to get container status \"a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7\": rpc error: code = NotFound desc = could not find container \"a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7\": container with ID starting with a713c5f595944ed852baa768c2dba0dbd30065aafcbdbe0f8e5b8ea9e7ba72e7 not found: ID does not exist" Nov 03 16:30:28.485198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.485178 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j982c"] Nov 03 16:30:28.489186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:28.489164 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j982c"] Nov 03 16:30:30.386601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:30.386558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:30.466055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:30.466026 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896bbdd8-02b1-47cc-9bdc-191c348be1e1" path="/var/lib/kubelet/pods/896bbdd8-02b1-47cc-9bdc-191c348be1e1/volumes" Nov 03 16:30:30.536827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:30.536802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:37.147719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.147688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:37.359782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.359743 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:37.818254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.818220 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98"] Nov 03 16:30:37.818693 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.818677 4539 cpu_manager.go:414] "containerMap: removing container" podUID="896bbdd8-02b1-47cc-9bdc-191c348be1e1" containerName="authorino" Nov 03 16:30:37.818749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.818695 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896bbdd8-02b1-47cc-9bdc-191c348be1e1" containerName="authorino" Nov 03 16:30:37.818795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.818763 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="896bbdd8-02b1-47cc-9bdc-191c348be1e1" containerName="authorino" Nov 03 16:30:37.821896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.821877 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:37.835320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.835300 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:30:37.835452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.835437 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:30:37.845650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.845630 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 16:30:37.845757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.845715 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-x4lc5\"" Nov 03 16:30:37.848606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.848583 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98"] Nov 03 16:30:37.925036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.925001 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae3fc69f-1702-4875-89d2-6ac92f1d5bd8-data\") pod \"postgres-7cdbf6dc44-87c98\" (UID: \"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:37.925206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.925047 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/ae3fc69f-1702-4875-89d2-6ac92f1d5bd8-kube-api-access-cr4xq\") pod \"postgres-7cdbf6dc44-87c98\" (UID: \"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:37.949987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:37.949953 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:38.026335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.026300 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae3fc69f-1702-4875-89d2-6ac92f1d5bd8-data\") pod \"postgres-7cdbf6dc44-87c98\" (UID: \"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:38.026576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.026348 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/ae3fc69f-1702-4875-89d2-6ac92f1d5bd8-kube-api-access-cr4xq\") pod \"postgres-7cdbf6dc44-87c98\" (UID: \"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:38.026776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.026752 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ae3fc69f-1702-4875-89d2-6ac92f1d5bd8-data\") pod \"postgres-7cdbf6dc44-87c98\" (UID: \"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:38.048859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.048824 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4xq\" (UniqueName: \"kubernetes.io/projected/ae3fc69f-1702-4875-89d2-6ac92f1d5bd8-kube-api-access-cr4xq\") pod \"postgres-7cdbf6dc44-87c98\" (UID: \"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:38.134538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.134454 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:38.431835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.431804 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98"] Nov 03 16:30:38.434567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:30:38.434536 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3fc69f_1702_4875_89d2_6ac92f1d5bd8.slice/crio-402301962a140d77d258a3b4f940ab9ebde160100c730d29533afbfbb2ffa9e4 WatchSource:0}: Error finding container 402301962a140d77d258a3b4f940ab9ebde160100c730d29533afbfbb2ffa9e4: Status 404 returned error can't find the container with id 402301962a140d77d258a3b4f940ab9ebde160100c730d29533afbfbb2ffa9e4 Nov 03 16:30:38.489651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:38.489620 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" event={"ID":"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8","Type":"ContainerStarted","Data":"402301962a140d77d258a3b4f940ab9ebde160100c730d29533afbfbb2ffa9e4"} Nov 03 16:30:43.522089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:43.521997 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" event={"ID":"ae3fc69f-1702-4875-89d2-6ac92f1d5bd8","Type":"ContainerStarted","Data":"d9e7b660fce5c5a72deb9663e6fb71ea737f10ded5c58e931ec8f427f93a6f71"} Nov 03 16:30:43.522494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:43.522149 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:43.561751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:43.561700 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" podStartSLOduration=1.8829135799999999 podStartE2EDuration="6.561687131s" totalImagesPullingTime="4.678773551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:43.559482352 +0000 UTC m=+987.252753376" watchObservedRunningTime="2026-11-03 16:30:43.561687131 +0000 UTC m=+987.254958158" Nov 03 16:30:49.556767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:49.556739 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-87c98" Nov 03 16:30:56.156782 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:30:56.156744 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:30:56.721749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:56.721711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:56.872140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:56.872099 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:57.018921 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:57.018841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:30:57.160494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:30:57.160458 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:00.350854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:00.350823 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:00.503080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:00.503038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:02.271338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:02.271308 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:02.420521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:02.420491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:08.402728 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:08.402693 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:19.441811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:19.441776 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:26.155495 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:31:26.155454 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:31:29.646170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:29.646093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:29.800178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:29.800139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:31.152306 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:31.152273 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:31.343205 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:31.343174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:31.551923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:31.551886 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:40.337851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.337816 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:40.596624 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.596537 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ccqgk"] Nov 03 16:31:40.600297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.600281 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:40.618293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.618266 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ccqgk"] Nov 03 16:31:40.625632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.625614 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvhsq\"" Nov 03 16:31:40.628168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.628136 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:31:40.777170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.777134 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447xz\" (UniqueName: \"kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz\") pod \"authorino-6f6944bcdd-ccqgk\" (UID: \"2658c9ab-be74-4e1b-9a22-bddc5d6e2134\") " pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:40.878322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.878226 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447xz\" (UniqueName: \"kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz\") pod \"authorino-6f6944bcdd-ccqgk\" (UID: \"2658c9ab-be74-4e1b-9a22-bddc5d6e2134\") " pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:40.900064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.900028 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447xz\" (UniqueName: \"kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz\") pod \"authorino-6f6944bcdd-ccqgk\" (UID: \"2658c9ab-be74-4e1b-9a22-bddc5d6e2134\") " pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:40.905217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.905188 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ccqgk"] Nov 03 16:31:40.905412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:40.905398 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:41.023908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.023875 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b774bd4dc-wjqvl"] Nov 03 16:31:41.027740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.027698 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.050330 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.050302 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b774bd4dc-wjqvl"] Nov 03 16:31:41.055177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.055153 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 16:31:41.083858 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.083833 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9b56\" (UniqueName: \"kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56\") pod \"authorino-5b774bd4dc-wjqvl\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.083976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.083895 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert\") pod \"authorino-5b774bd4dc-wjqvl\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.184851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.184762 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert\") pod \"authorino-5b774bd4dc-wjqvl\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.185016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.184866 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9b56\" (UniqueName: \"kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56\") pod \"authorino-5b774bd4dc-wjqvl\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.187291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.187272 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert\") pod \"authorino-5b774bd4dc-wjqvl\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.206136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.206114 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9b56\" (UniqueName: \"kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56\") pod \"authorino-5b774bd4dc-wjqvl\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.216521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.216500 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ccqgk"] Nov 03 16:31:41.218594 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:31:41.218568 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2658c9ab_be74_4e1b_9a22_bddc5d6e2134.slice/crio-33f61e8489224a13b00b78e3a21ae0ee3caf78ed446cdb49bde988141f3cbfe3 WatchSource:0}: Error finding container 33f61e8489224a13b00b78e3a21ae0ee3caf78ed446cdb49bde988141f3cbfe3: Status 404 returned error can't find the container with id 33f61e8489224a13b00b78e3a21ae0ee3caf78ed446cdb49bde988141f3cbfe3 Nov 03 16:31:41.339823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.339786 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:31:41.645666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.639901 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b774bd4dc-wjqvl"] Nov 03 16:31:41.809296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.809243 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" event={"ID":"2658c9ab-be74-4e1b-9a22-bddc5d6e2134","Type":"ContainerStarted","Data":"f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c"} Nov 03 16:31:41.809296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.809297 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" event={"ID":"2658c9ab-be74-4e1b-9a22-bddc5d6e2134","Type":"ContainerStarted","Data":"33f61e8489224a13b00b78e3a21ae0ee3caf78ed446cdb49bde988141f3cbfe3"} Nov 03 16:31:41.809600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.809260 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" podUID="2658c9ab-be74-4e1b-9a22-bddc5d6e2134" containerName="authorino" containerID="cri-o://f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c" gracePeriod=30 Nov 03 16:31:41.811060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.811035 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" event={"ID":"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3","Type":"ContainerStarted","Data":"263d3589f57c5889fdcc9b9156eb4367b846a84caabd0294c0f886d85fa12f3e"} Nov 03 16:31:41.850232 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:41.850176 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" podStartSLOduration=1.342688699 podStartE2EDuration="1.850161957s" totalImagesPullingTime="507.473258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:41.849506787 +0000 UTC m=+1045.542777808" watchObservedRunningTime="2026-11-03 16:31:41.850161957 +0000 UTC m=+1045.543433008" Nov 03 16:31:42.078765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.076959 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:42.096868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.096828 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz\" (UniqueName: \"kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz\") pod \"2658c9ab-be74-4e1b-9a22-bddc5d6e2134\" (UID: \"2658c9ab-be74-4e1b-9a22-bddc5d6e2134\") " Nov 03 16:31:42.099067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.099042 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz" pod "2658c9ab-be74-4e1b-9a22-bddc5d6e2134" (UID: "2658c9ab-be74-4e1b-9a22-bddc5d6e2134"). InnerVolumeSpecName "kube-api-access-447xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:42.198001 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.197969 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447xz\" (UniqueName: \"kubernetes.io/projected/2658c9ab-be74-4e1b-9a22-bddc5d6e2134-kube-api-access-447xz\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:31:42.818697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.818662 4539 generic.go:356] "Generic (PLEG): container finished" podID="2658c9ab-be74-4e1b-9a22-bddc5d6e2134" containerID="f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c" exitCode=2 Nov 03 16:31:42.818697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.818711 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" Nov 03 16:31:42.819236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.818741 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" event={"ID":"2658c9ab-be74-4e1b-9a22-bddc5d6e2134","Type":"ContainerDied","Data":"f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c"} Nov 03 16:31:42.819236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.818795 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ccqgk" event={"ID":"2658c9ab-be74-4e1b-9a22-bddc5d6e2134","Type":"ContainerDied","Data":"33f61e8489224a13b00b78e3a21ae0ee3caf78ed446cdb49bde988141f3cbfe3"} Nov 03 16:31:42.819236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.818814 4539 scope.go:122] "RemoveContainer" containerID="f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c" Nov 03 16:31:42.821379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.820914 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" event={"ID":"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3","Type":"ContainerStarted","Data":"b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9"} Nov 03 16:31:42.823036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.823011 4539 generic.go:356] "Generic (PLEG): container finished" podID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerID="72ff845877700413c53a4664fc691d7d224f85211bebb9adc16b42ad9dc505f2" exitCode=0 Nov 03 16:31:42.823145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.823090 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" event={"ID":"40e8a12c-f0a1-4690-ab57-56958719f4ba","Type":"ContainerDied","Data":"72ff845877700413c53a4664fc691d7d224f85211bebb9adc16b42ad9dc505f2"} Nov 03 16:31:42.823584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.823505 4539 scope.go:122] "RemoveContainer" containerID="72ff845877700413c53a4664fc691d7d224f85211bebb9adc16b42ad9dc505f2" Nov 03 16:31:42.824841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.824792 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:31:42.824954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.824874 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:31:42.824954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.824906 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:31:42.830233 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.830214 4539 scope.go:122] "RemoveContainer" containerID="f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c" Nov 03 16:31:42.830533 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:31:42.830512 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c\": container with ID starting with f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c not found: ID does not exist" containerID="f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c" Nov 03 16:31:42.830604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.830544 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c"} err="failed to get container status \"f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c\": rpc error: code = NotFound desc = could not find container \"f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c\": container with ID starting with f5d8fc1ade61cf9c30097bc054e8bf86af94321088af9121b9bb102a52a9082c not found: ID does not exist" Nov 03 16:31:42.873237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.873179 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" podStartSLOduration=2.542717745 podStartE2EDuration="2.873154114s" totalImagesPullingTime="330.436369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:42.872730919 +0000 UTC m=+1046.566001941" watchObservedRunningTime="2026-11-03 16:31:42.873154114 +0000 UTC m=+1046.566425137" Nov 03 16:31:42.955403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.955334 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ccqgk"] Nov 03 16:31:42.959357 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:42.959317 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ccqgk"] Nov 03 16:31:43.832976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.832941 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" event={"ID":"40e8a12c-f0a1-4690-ab57-56958719f4ba","Type":"ContainerStarted","Data":"dae5b5e676ee7b6b3785457e1529d1e00fb84aa4fe85be07ede488404f796730"} Nov 03 16:31:43.927917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.927872 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv"] Nov 03 16:31:43.928534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.928513 4539 cpu_manager.go:414] "containerMap: removing container" podUID="2658c9ab-be74-4e1b-9a22-bddc5d6e2134" containerName="authorino" Nov 03 16:31:43.928632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.928536 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2658c9ab-be74-4e1b-9a22-bddc5d6e2134" containerName="authorino" Nov 03 16:31:43.928696 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.928642 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="2658c9ab-be74-4e1b-9a22-bddc5d6e2134" containerName="authorino" Nov 03 16:31:43.932074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.932049 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:43.934933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:43.934907 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv"] Nov 03 16:31:44.030963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.030912 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkcf\" (UniqueName: \"kubernetes.io/projected/a88f549e-c107-4cb0-a806-5b9dcc15c271-kube-api-access-4vkcf\") pod \"llmisvc-controller-manager-6466c554d8-2bgbv\" (UID: \"a88f549e-c107-4cb0-a806-5b9dcc15c271\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.030963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.030968 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88f549e-c107-4cb0-a806-5b9dcc15c271-cert\") pod \"llmisvc-controller-manager-6466c554d8-2bgbv\" (UID: \"a88f549e-c107-4cb0-a806-5b9dcc15c271\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.131528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.131414 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88f549e-c107-4cb0-a806-5b9dcc15c271-cert\") pod \"llmisvc-controller-manager-6466c554d8-2bgbv\" (UID: \"a88f549e-c107-4cb0-a806-5b9dcc15c271\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.131688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.131581 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkcf\" (UniqueName: \"kubernetes.io/projected/a88f549e-c107-4cb0-a806-5b9dcc15c271-kube-api-access-4vkcf\") pod \"llmisvc-controller-manager-6466c554d8-2bgbv\" (UID: \"a88f549e-c107-4cb0-a806-5b9dcc15c271\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.134085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.134057 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88f549e-c107-4cb0-a806-5b9dcc15c271-cert\") pod \"llmisvc-controller-manager-6466c554d8-2bgbv\" (UID: \"a88f549e-c107-4cb0-a806-5b9dcc15c271\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.153612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.153571 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkcf\" (UniqueName: \"kubernetes.io/projected/a88f549e-c107-4cb0-a806-5b9dcc15c271-kube-api-access-4vkcf\") pod \"llmisvc-controller-manager-6466c554d8-2bgbv\" (UID: \"a88f549e-c107-4cb0-a806-5b9dcc15c271\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.247063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.247026 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:44.467336 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.467254 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2658c9ab-be74-4e1b-9a22-bddc5d6e2134" path="/var/lib/kubelet/pods/2658c9ab-be74-4e1b-9a22-bddc5d6e2134/volumes" Nov 03 16:31:44.468575 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.468556 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:31:44.469752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.469730 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 03 16:31:44.469834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.469779 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 03 16:31:44.547174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.547146 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv"] Nov 03 16:31:44.552121 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:31:44.552092 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88f549e_c107_4cb0_a806_5b9dcc15c271.slice/crio-0a92ae7f8d18cc7030f2035168ff19644e8499ca2f9f045234ae485dbf1bbac6 WatchSource:0}: Error finding container 0a92ae7f8d18cc7030f2035168ff19644e8499ca2f9f045234ae485dbf1bbac6: Status 404 returned error can't find the container with id 0a92ae7f8d18cc7030f2035168ff19644e8499ca2f9f045234ae485dbf1bbac6 Nov 03 16:31:44.839391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:44.839358 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" event={"ID":"a88f549e-c107-4cb0-a806-5b9dcc15c271","Type":"ContainerStarted","Data":"0a92ae7f8d18cc7030f2035168ff19644e8499ca2f9f045234ae485dbf1bbac6"} Nov 03 16:31:45.468703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:45.468660 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 03 16:31:45.468881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:45.468723 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 03 16:31:45.846745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:45.846701 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" event={"ID":"a88f549e-c107-4cb0-a806-5b9dcc15c271","Type":"ContainerStarted","Data":"8251d4f1b287b0dcf436aa41e3862239c81c43f8268dc7f5a27fe5669ad1770b"} Nov 03 16:31:45.847152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:45.846806 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:31:45.887143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:45.887093 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" podStartSLOduration=2.586509154 podStartE2EDuration="2.88707688s" totalImagesPullingTime="300.567726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:45.885174285 +0000 UTC m=+1049.578445308" watchObservedRunningTime="2026-11-03 16:31:45.88707688 +0000 UTC m=+1049.580347903" Nov 03 16:31:46.468810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:46.468780 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 03 16:31:46.468978 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:46.468827 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 03 16:31:47.469315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:47.469280 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 03 16:31:47.469821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:47.469335 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 03 16:31:47.693637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:47.693600 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:31:48.468630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:48.468598 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Nov 03 16:31:48.468806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:48.468654 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Nov 03 16:31:49.472751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:49.472722 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:31:49.473650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:31:49.473633 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:31:56.156614 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:31:56.156574 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:32:08.503194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:08.503158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:16.852185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:16.852082 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2bgbv" Nov 03 16:32:16.919057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:16.919027 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72"] Nov 03 16:32:16.919380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:16.919208 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" podUID="b7e98e62-20a2-462f-a268-342dd93bf569" containerName="manager" containerID="cri-o://91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa" gracePeriod=30 Nov 03 16:32:17.170815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.170793 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:32:17.315153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.315119 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l\" (UniqueName: \"kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l\") pod \"b7e98e62-20a2-462f-a268-342dd93bf569\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " Nov 03 16:32:17.315339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.315191 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert\" (UniqueName: \"kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert\") pod \"b7e98e62-20a2-462f-a268-342dd93bf569\" (UID: \"b7e98e62-20a2-462f-a268-342dd93bf569\") " Nov 03 16:32:17.317439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.317403 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l" pod "b7e98e62-20a2-462f-a268-342dd93bf569" (UID: "b7e98e62-20a2-462f-a268-342dd93bf569"). InnerVolumeSpecName "kube-api-access-rls5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:17.317439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.317410 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert" pod "b7e98e62-20a2-462f-a268-342dd93bf569" (UID: "b7e98e62-20a2-462f-a268-342dd93bf569"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:17.415919 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.415841 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rls5l\" (UniqueName: \"kubernetes.io/projected/b7e98e62-20a2-462f-a268-342dd93bf569-kube-api-access-rls5l\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:32:17.415919 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:17.415865 4539 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7e98e62-20a2-462f-a268-342dd93bf569-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:32:18.017222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.017187 4539 generic.go:356] "Generic (PLEG): container finished" podID="b7e98e62-20a2-462f-a268-342dd93bf569" containerID="91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa" exitCode=0 Nov 03 16:32:18.017726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.017235 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" event={"ID":"b7e98e62-20a2-462f-a268-342dd93bf569","Type":"ContainerDied","Data":"91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa"} Nov 03 16:32:18.017726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.017253 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" Nov 03 16:32:18.017726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.017297 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72" event={"ID":"b7e98e62-20a2-462f-a268-342dd93bf569","Type":"ContainerDied","Data":"50f102ca5ddb5a9dc6c1fdd7e17795a49a6cc79abcad1006fa07c6b66dcdfb93"} Nov 03 16:32:18.017726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.017318 4539 scope.go:122] "RemoveContainer" containerID="91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa" Nov 03 16:32:18.026632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.026616 4539 scope.go:122] "RemoveContainer" containerID="91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa" Nov 03 16:32:18.026900 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:32:18.026882 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa\": container with ID starting with 91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa not found: ID does not exist" containerID="91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa" Nov 03 16:32:18.026959 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.026909 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa"} err="failed to get container status \"91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa\": rpc error: code = NotFound desc = could not find container \"91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa\": container with ID starting with 91a71d27439074cee2e145eca0a2eae73968764f7513d4d1567fafbd8df597aa not found: ID does not exist" Nov 03 16:32:18.063742 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.063720 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72"] Nov 03 16:32:18.073868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.073845 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dmg72"] Nov 03 16:32:18.466919 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.466888 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e98e62-20a2-462f-a268-342dd93bf569" path="/var/lib/kubelet/pods/b7e98e62-20a2-462f-a268-342dd93bf569/volumes" Nov 03 16:32:18.540156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.540126 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:18.758773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:18.758677 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:22.000883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:22.000840 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:22.169898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:22.169870 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:23.571531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.571504 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629"] Nov 03 16:32:23.571918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.571849 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b7e98e62-20a2-462f-a268-342dd93bf569" containerName="manager" Nov 03 16:32:23.571918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.571860 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e98e62-20a2-462f-a268-342dd93bf569" containerName="manager" Nov 03 16:32:23.571992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.571936 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e98e62-20a2-462f-a268-342dd93bf569" containerName="manager" Nov 03 16:32:23.575786 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.575766 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.608039 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.608008 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629"] Nov 03 16:32:23.669080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669046 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669123 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669169 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669294 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669329 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669418 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669469 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2sbp\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669575 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669523 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.669575 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.669556 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770161 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770205 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770255 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770272 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770303 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770326 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770380 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770408 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2sbp\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770467 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770687 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770780 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770848 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.770912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.770864 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.771239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.771221 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.772908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.772887 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.773122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.773100 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.792373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.792351 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.792551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.792535 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2sbp\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tw629\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:23.888929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:23.888853 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:24.202095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:24.202072 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629"] Nov 03 16:32:24.203592 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:32:24.203569 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eaf3234_d440_4b4d_874b_28235bcb8cac.slice/crio-a0a0590522e6b4b601766eb02e6f8922c400c042f80821081462a1ca492cd325 WatchSource:0}: Error finding container a0a0590522e6b4b601766eb02e6f8922c400c042f80821081462a1ca492cd325: Status 404 returned error can't find the container with id a0a0590522e6b4b601766eb02e6f8922c400c042f80821081462a1ca492cd325 Nov 03 16:32:24.205622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:24.205591 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:32:24.205700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:24.205658 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:32:24.205700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:24.205692 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:32:25.061223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:25.061190 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" event={"ID":"9eaf3234-d440-4b4d-874b-28235bcb8cac","Type":"ContainerStarted","Data":"b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2"} Nov 03 16:32:25.061636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:25.061239 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" event={"ID":"9eaf3234-d440-4b4d-874b-28235bcb8cac","Type":"ContainerStarted","Data":"a0a0590522e6b4b601766eb02e6f8922c400c042f80821081462a1ca492cd325"} Nov 03 16:32:25.889470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:25.889408 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:25.890848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:25.890820 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tw629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:32:25.891008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:25.890887 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 16:32:26.156158 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:32:26.156072 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:32:26.890076 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:26.890042 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tw629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:32:26.890249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:26.890106 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 16:32:27.890092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:27.890057 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tw629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:32:27.890511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:27.890109 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 16:32:28.889272 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:28.889238 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tw629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:32:28.889483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:28.889289 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 16:32:29.889634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:29.889598 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tw629 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:32:29.890029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:29.889649 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 16:32:30.893117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:30.893083 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:30.893538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:30.893356 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:30.893990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:30.893970 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:32:30.970715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:30.970663 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podStartSLOduration=7.970649302 podStartE2EDuration="7.970649302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:25.121730928 +0000 UTC m=+1088.815001954" watchObservedRunningTime="2026-11-03 16:32:30.970649302 +0000 UTC m=+1094.663920323" Nov 03 16:32:38.604487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.604451 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn"] Nov 03 16:32:38.607956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.607937 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.623751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.623719 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.623751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.623754 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.623945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.623779 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.623945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.623865 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.623945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.623914 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c46d2b-77a9-4239-a40c-2b2152f04e00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.624083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.623981 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.624083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.624013 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.624083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.624028 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.624083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.624050 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k54\" (UniqueName: \"kubernetes.io/projected/74c46d2b-77a9-4239-a40c-2b2152f04e00-kube-api-access-54k54\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.651174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.651141 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn"] Nov 03 16:32:38.724946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.724913 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.724946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.724947 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.724977 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725082 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725131 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c46d2b-77a9-4239-a40c-2b2152f04e00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725244 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725325 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725353 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725392 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k54\" (UniqueName: \"kubernetes.io/projected/74c46d2b-77a9-4239-a40c-2b2152f04e00-kube-api-access-54k54\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725405 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725477 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725708 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725777 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.725860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.725795 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c46d2b-77a9-4239-a40c-2b2152f04e00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.727644 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.727621 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.727754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.727736 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.747698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.747661 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c46d2b-77a9-4239-a40c-2b2152f04e00-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.748930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.748905 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k54\" (UniqueName: \"kubernetes.io/projected/74c46d2b-77a9-4239-a40c-2b2152f04e00-kube-api-access-54k54\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zvhjn\" (UID: \"74c46d2b-77a9-4239-a40c-2b2152f04e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:38.922163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:38.922063 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:39.236143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:39.236113 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn"] Nov 03 16:32:39.237860 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:32:39.237834 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c46d2b_77a9_4239_a40c_2b2152f04e00.slice/crio-34a5a6efdad52016872a75f77a833e7424c723e66221e6aec9f44b087cd71201 WatchSource:0}: Error finding container 34a5a6efdad52016872a75f77a833e7424c723e66221e6aec9f44b087cd71201: Status 404 returned error can't find the container with id 34a5a6efdad52016872a75f77a833e7424c723e66221e6aec9f44b087cd71201 Nov 03 16:32:39.240213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:39.240176 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:32:39.240291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:39.240263 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:32:39.240328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:39.240304 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:32:40.138845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:40.138710 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" event={"ID":"74c46d2b-77a9-4239-a40c-2b2152f04e00","Type":"ContainerStarted","Data":"3605be3c349e1a68f6fcc95f03a58d30793f01e8dd6d354d8123eb12b58905ca"} Nov 03 16:32:40.139450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:40.138873 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" event={"ID":"74c46d2b-77a9-4239-a40c-2b2152f04e00","Type":"ContainerStarted","Data":"34a5a6efdad52016872a75f77a833e7424c723e66221e6aec9f44b087cd71201"} Nov 03 16:32:40.225448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:40.225320 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podStartSLOduration=2.225297814 podStartE2EDuration="2.225297814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:40.197066909 +0000 UTC m=+1103.890337931" watchObservedRunningTime="2026-11-03 16:32:40.225297814 +0000 UTC m=+1103.918568838" Nov 03 16:32:40.922639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:40.922601 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:40.925044 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:40.925007 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:32:40.925210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:40.925076 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podUID="74c46d2b-77a9-4239-a40c-2b2152f04e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 16:32:41.924889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:41.924850 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:32:41.925283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:41.924919 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podUID="74c46d2b-77a9-4239-a40c-2b2152f04e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 16:32:42.924864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:42.924829 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:32:42.925046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:42.924887 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podUID="74c46d2b-77a9-4239-a40c-2b2152f04e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 16:32:43.922914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:43.922873 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:32:43.923268 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:43.923226 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podUID="74c46d2b-77a9-4239-a40c-2b2152f04e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 16:32:44.922908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:44.922865 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 16:32:44.923976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:44.923555 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podUID="74c46d2b-77a9-4239-a40c-2b2152f04e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 16:32:45.926249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:45.926212 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:45.926706 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:45.926630 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:45.927224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:45.927207 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" Nov 03 16:32:46.895544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:46.895510 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:47.419543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:47.419509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:48.704368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:48.704334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:48.855849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:48.855805 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:52.966688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:52.966645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:56.157023 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:32:56.156984 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:32:56.891836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:56.891795 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:32:58.341516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:32:58.341470 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:00.388469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:00.387595 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:02.978472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:02.978418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:06.230278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:06.230239 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:08.613739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:08.613697 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:10.893888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:10.893844 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:12.860365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:12.860328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:26.157454 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:33:26.157405 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:33:31.196821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:31.196786 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:31.345769 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:31.345736 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:33.105318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:33.105286 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:33.274339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:33.274305 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:33.987291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:33.987256 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b774bd4dc-wjqvl"] Nov 03 16:33:33.987538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:33.987512 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" podUID="170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" containerName="authorino" containerID="cri-o://b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9" gracePeriod=30 Nov 03 16:33:34.248160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.248088 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:33:34.380131 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.380102 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert\" (UniqueName: \"kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert\") pod \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " Nov 03 16:33:34.380328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.380209 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56\" (UniqueName: \"kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56\") pod \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\" (UID: \"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3\") " Nov 03 16:33:34.382489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.382453 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56" pod "170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" (UID: "170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3"). InnerVolumeSpecName "kube-api-access-c9b56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:33:34.391049 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.391019 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert" pod "170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" (UID: "170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:33:34.481210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.481178 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9b56\" (UniqueName: \"kubernetes.io/projected/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-kube-api-access-c9b56\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:33:34.481210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.481206 4539 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3-tls-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:33:34.548169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.548129 4539 generic.go:356] "Generic (PLEG): container finished" podID="170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" containerID="b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9" exitCode=0 Nov 03 16:33:34.548169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.548169 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" Nov 03 16:33:34.548519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.548206 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" event={"ID":"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3","Type":"ContainerDied","Data":"b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9"} Nov 03 16:33:34.548519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.548257 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b774bd4dc-wjqvl" event={"ID":"170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3","Type":"ContainerDied","Data":"263d3589f57c5889fdcc9b9156eb4367b846a84caabd0294c0f886d85fa12f3e"} Nov 03 16:33:34.548519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.548281 4539 scope.go:122] "RemoveContainer" containerID="b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9" Nov 03 16:33:34.560453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.560419 4539 scope.go:122] "RemoveContainer" containerID="b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9" Nov 03 16:33:34.560793 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:33:34.560764 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9\": container with ID starting with b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9 not found: ID does not exist" containerID="b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9" Nov 03 16:33:34.560879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.560805 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9"} err="failed to get container status \"b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9\": rpc error: code = NotFound desc = could not find container \"b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9\": container with ID starting with b2676aaaa4e3e419c6d5b5ee7f7e1022dc8cf4837f6b615c78d3072a22e19fc9 not found: ID does not exist" Nov 03 16:33:34.599605 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.599568 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b774bd4dc-wjqvl"] Nov 03 16:33:34.617926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.617884 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b774bd4dc-wjqvl"] Nov 03 16:33:34.695000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.694967 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:34.844398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:34.844306 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:33:36.474054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:33:36.469873 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" path="/var/lib/kubelet/pods/170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3/volumes" Nov 03 16:33:56.155762 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:33:56.155719 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:34:05.750268 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:05.750232 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:05.915989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:05.915954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:06.064773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:06.064736 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:06.259618 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:06.259584 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:06.441148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:06.441061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:06.610703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:06.610670 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:06.780137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:06.780088 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:07.043223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:07.043138 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:07.191369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:07.191339 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:07.636870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:07.636841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:08.120935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:08.120905 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:08.292610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:08.292566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:08.719963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:08.719926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:09.410927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:09.410894 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:09.614081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:09.614041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:09.809616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:09.809569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:10.072676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:10.072565 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:10.228812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:10.228781 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:10.390784 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:10.390692 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:10.942471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:10.942409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:11.104153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:11.104121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:11.657579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:11.657550 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:12.011489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:12.011380 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:12.213120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:12.213086 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:12.812210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:12.812173 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:12.978418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:12.978345 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:13.139727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.139694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:13.367156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.367122 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:13.780903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.780876 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:13.924600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.924570 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf"] Nov 03 16:34:13.924974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.924962 4539 cpu_manager.go:414] "containerMap: removing container" podUID="170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" containerName="authorino" Nov 03 16:34:13.925017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.924975 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" containerName="authorino" Nov 03 16:34:13.925055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.925034 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="170cfdcc-7fe5-4bf1-a2b2-d6af1d344bf3" containerName="authorino" Nov 03 16:34:13.928325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.928299 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:13.948835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.948803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:13.953104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.953078 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c9fa1268-dockercfg-4g2d5\"" Nov 03 16:34:13.953264 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.953178 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c9fa1268\"" Nov 03 16:34:13.956292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:13.956268 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf"] Nov 03 16:34:14.042537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.042456 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.042690 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.042552 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtfm\" (UniqueName: \"kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm\") pod \"payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.143232 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.143198 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.143677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.143256 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtfm\" (UniqueName: \"kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm\") pod \"payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.143895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.143875 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.165091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.165061 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtfm\" (UniqueName: \"kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm\") pod \"payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.243254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.243216 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:14.547713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.547680 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf"] Nov 03 16:34:14.550144 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:34:14.550112 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1882612_d579_48ef_9ad8_d7018e81b1f3.slice/crio-ede4e262f3e20bcf944ed1e14dd9801e79ba39e6d6ddccb3b06643e37eacb7f6 WatchSource:0}: Error finding container ede4e262f3e20bcf944ed1e14dd9801e79ba39e6d6ddccb3b06643e37eacb7f6: Status 404 returned error can't find the container with id ede4e262f3e20bcf944ed1e14dd9801e79ba39e6d6ddccb3b06643e37eacb7f6 Nov 03 16:34:14.642785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.642751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:14.762293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.762260 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" event={"ID":"c1882612-d579-48ef-9ad8-d7018e81b1f3","Type":"ContainerStarted","Data":"ede4e262f3e20bcf944ed1e14dd9801e79ba39e6d6ddccb3b06643e37eacb7f6"} Nov 03 16:34:14.815680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.815602 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:14.976226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:14.976179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:15.129368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.129255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:15.298554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.298524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:15.449057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.448971 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:15.832990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.832954 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh"] Nov 03 16:34:15.840331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.840305 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.867667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.867634 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4d5492de-openshift-default-dockercfg-bxx25\"" Nov 03 16:34:15.878533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.878511 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh"] Nov 03 16:34:15.964941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.964906 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.965208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.965186 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrpw\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.965340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.965325 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.965488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.965471 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.985543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.982771 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.985543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.982967 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.985543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.983047 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.985543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.983121 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:15.985543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:15.983252 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.083952 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.083994 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084019 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084082 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084125 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084182 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrpw\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084221 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084252 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084303 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084464 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084553 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.084780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.084759 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.085375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.085292 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.086026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.086002 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.087319 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.087297 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.091640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.091599 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:16.092889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.092867 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.113608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.113539 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrpw\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.114693 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.114669 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token\") pod \"e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.169927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.169894 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:16.250019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.249939 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:16.503065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.502869 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh"] Nov 03 16:34:16.509071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.509036 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:34:16.509166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.509118 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:34:16.509210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.509162 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:34:16.605163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.605138 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:34:16.607108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.607083 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:34:16.782711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.782667 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" event={"ID":"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c","Type":"ContainerStarted","Data":"da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520"} Nov 03 16:34:16.782882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.782772 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" event={"ID":"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c","Type":"ContainerStarted","Data":"e963a4ce2505e08d619135d3847a961957650f3df10281a339e0c293e8a7cc27"} Nov 03 16:34:16.833846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.833796 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podStartSLOduration=1.833778788 podStartE2EDuration="1.833778788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:16.832797272 +0000 UTC m=+1200.526068298" watchObservedRunningTime="2026-11-03 16:34:16.833778788 +0000 UTC m=+1200.527049810" Nov 03 16:34:16.887447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:16.887390 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:17.063930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.063843 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:17.170083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.170050 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:17.175399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.175377 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:17.260590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.260558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:17.790162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.790119 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" event={"ID":"c1882612-d579-48ef-9ad8-d7018e81b1f3","Type":"ContainerStarted","Data":"1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c"} Nov 03 16:34:17.790658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.790321 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:17.790658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.790370 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:17.791217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.791201 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:17.834224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:17.834161 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" podStartSLOduration=2.502009114 podStartE2EDuration="4.834141485s" totalImagesPullingTime="2.332132371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:17.830987474 +0000 UTC m=+1201.524258496" watchObservedRunningTime="2026-11-03 16:34:17.834141485 +0000 UTC m=+1201.527412509" Nov 03 16:34:18.580211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:18.580180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:18.749205 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:18.749171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:18.905171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:18.905071 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:20.038648 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.038614 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:20.501399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.501316 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:20.652035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.652005 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:20.671987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.671953 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f"] Nov 03 16:34:20.675937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.675913 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.698928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.698906 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4d5492de-dockercfg-2dj8r\"" Nov 03 16:34:20.702281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.702251 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f"] Nov 03 16:34:20.737148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.737115 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.737316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.737287 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dttb\" (UniqueName: \"kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb\") pod \"payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.744547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.744513 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4d5492de\"" Nov 03 16:34:20.745010 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.744988 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms"] Nov 03 16:34:20.749412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.749373 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.775946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.775918 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms"] Nov 03 16:34:20.837990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.837950 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume\") pod \"payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.838142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.838089 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dttb\" (UniqueName: \"kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb\") pod \"payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.838142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.838123 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wj68\" (UniqueName: \"kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68\") pod \"payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.838243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.838196 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.839042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.839021 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.860633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.860609 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dttb\" (UniqueName: \"kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb\") pod \"payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:20.938715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.938682 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wj68\" (UniqueName: \"kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68\") pod \"payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.938855 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.938759 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume\") pod \"payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.939662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.939643 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume\") pod \"payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.960349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.960311 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wj68\" (UniqueName: \"kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68\") pod \"payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:20.990058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:20.990038 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:21.082833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.082797 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:21.254943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.254912 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:21.290105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.290073 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f"] Nov 03 16:34:21.292450 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:34:21.292372 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02398b84_8f13_4118_95e4_87cd66678884.slice/crio-4403d998bd7bedea4c2755d0f970f8c3462159c9f71c3f29c4cd322e2f5caec2 WatchSource:0}: Error finding container 4403d998bd7bedea4c2755d0f970f8c3462159c9f71c3f29c4cd322e2f5caec2: Status 404 returned error can't find the container with id 4403d998bd7bedea4c2755d0f970f8c3462159c9f71c3f29c4cd322e2f5caec2 Nov 03 16:34:21.385468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.385413 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms"] Nov 03 16:34:21.387442 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:34:21.387403 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0610cd84_1893_46b0_ab15_5186095b8d39.slice/crio-d724ccf08fa7669238429ce59b4057670814beed8083e4f77aa0956b9315a41d WatchSource:0}: Error finding container d724ccf08fa7669238429ce59b4057670814beed8083e4f77aa0956b9315a41d: Status 404 returned error can't find the container with id d724ccf08fa7669238429ce59b4057670814beed8083e4f77aa0956b9315a41d Nov 03 16:34:21.818805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.818769 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" event={"ID":"0610cd84-1893-46b0-ab15-5186095b8d39","Type":"ContainerStarted","Data":"27e71c420275cdc2aee43fe045d6a7e10addbf84947405723b4dfcd8a7432b28"} Nov 03 16:34:21.819016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.818819 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" event={"ID":"0610cd84-1893-46b0-ab15-5186095b8d39","Type":"ContainerStarted","Data":"d724ccf08fa7669238429ce59b4057670814beed8083e4f77aa0956b9315a41d"} Nov 03 16:34:21.819016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.818871 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:21.820620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.820592 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" event={"ID":"02398b84-8f13-4118-95e4-87cd66678884","Type":"ContainerStarted","Data":"ffccd929327f49f7f44d3285211896fdb1a680317331ab93edf6bcd2a6d0662b"} Nov 03 16:34:21.820763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.820632 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" event={"ID":"02398b84-8f13-4118-95e4-87cd66678884","Type":"ContainerStarted","Data":"4403d998bd7bedea4c2755d0f970f8c3462159c9f71c3f29c4cd322e2f5caec2"} Nov 03 16:34:21.820763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.820673 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:21.858073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.858015 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" podStartSLOduration=1.858001597 podStartE2EDuration="1.858001597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:21.857007215 +0000 UTC m=+1205.550278238" watchObservedRunningTime="2026-11-03 16:34:21.858001597 +0000 UTC m=+1205.551272647" Nov 03 16:34:21.906698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:21.906648 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" podStartSLOduration=1.906634132 podStartE2EDuration="1.906634132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:21.905987691 +0000 UTC m=+1205.599258713" watchObservedRunningTime="2026-11-03 16:34:21.906634132 +0000 UTC m=+1205.599905153" Nov 03 16:34:22.985599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:22.985492 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:24.814284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:24.814249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:26.156563 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:34:26.156527 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:34:26.835210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:26.835174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:28.469055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:28.468986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:28.628011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:28.627981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:28.772193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:28.772116 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:28.799509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:28.799478 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:29.610986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:29.610955 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:30.568985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.568955 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj"] Nov 03 16:34:30.572466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.572447 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.586488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.586469 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:34:30.596351 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.596330 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:34:30.598860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.598838 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj"] Nov 03 16:34:30.632121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.632089 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.632439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.632129 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w454w\" (UniqueName: \"kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.632439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.632280 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.670929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.670901 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:30.733729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.733691 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.733993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.733961 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w454w\" (UniqueName: \"kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.734125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.734084 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.734562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.734533 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.736726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.736703 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.756954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.756922 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w454w\" (UniqueName: \"kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w\") pod \"maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:30.885574 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:30.885500 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:31.183582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:31.183561 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj"] Nov 03 16:34:31.185731 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:34:31.185701 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8113cd_2845_4279_929b_45ef4f8c798b.slice/crio-70c7138e055d3d29144b4812c82dad1198a531fdeb24d1a875c9047006519179 WatchSource:0}: Error finding container 70c7138e055d3d29144b4812c82dad1198a531fdeb24d1a875c9047006519179: Status 404 returned error can't find the container with id 70c7138e055d3d29144b4812c82dad1198a531fdeb24d1a875c9047006519179 Nov 03 16:34:31.882994 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:31.882948 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" event={"ID":"2e8113cd-2845-4279-929b-45ef4f8c798b","Type":"ContainerStarted","Data":"70c7138e055d3d29144b4812c82dad1198a531fdeb24d1a875c9047006519179"} Nov 03 16:34:32.665876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:32.665833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:32.830491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:32.829828 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:32.838392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:32.838359 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:34.418449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:34.415307 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:34.910352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:34.909514 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" event={"ID":"2e8113cd-2845-4279-929b-45ef4f8c798b","Type":"ContainerStarted","Data":"a52c958d48644213229ce68960b16994d514778690eff556876e70f278da2f1c"} Nov 03 16:34:34.952161 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:34.952112 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" podStartSLOduration=2.12237743 podStartE2EDuration="4.952096924s" totalImagesPullingTime="2.829719494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:34.951449043 +0000 UTC m=+1218.644720060" watchObservedRunningTime="2026-11-03 16:34:34.952096924 +0000 UTC m=+1218.645367946" Nov 03 16:34:35.575173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:35.575145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:35.733490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:35.733438 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:35.932580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:35.932474 4539 generic.go:356] "Generic (PLEG): container finished" podID="2e8113cd-2845-4279-929b-45ef4f8c798b" containerID="a52c958d48644213229ce68960b16994d514778690eff556876e70f278da2f1c" exitCode=0 Nov 03 16:34:35.932580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:35.932542 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" event={"ID":"2e8113cd-2845-4279-929b-45ef4f8c798b","Type":"ContainerDied","Data":"a52c958d48644213229ce68960b16994d514778690eff556876e70f278da2f1c"} Nov 03 16:34:36.075902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:36.075862 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:37.094062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.094027 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:37.194297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.194215 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca\") pod \"2e8113cd-2845-4279-929b-45ef4f8c798b\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " Nov 03 16:34:37.194485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.194340 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca\") pod \"2e8113cd-2845-4279-929b-45ef4f8c798b\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " Nov 03 16:34:37.194485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.194388 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w\" (UniqueName: \"kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w\") pod \"2e8113cd-2845-4279-929b-45ef4f8c798b\" (UID: \"2e8113cd-2845-4279-929b-45ef4f8c798b\") " Nov 03 16:34:37.194727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.194658 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca" pod "2e8113cd-2845-4279-929b-45ef4f8c798b" (UID: "2e8113cd-2845-4279-929b-45ef4f8c798b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:37.196740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.196709 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca" pod "2e8113cd-2845-4279-929b-45ef4f8c798b" (UID: "2e8113cd-2845-4279-929b-45ef4f8c798b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:37.196852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.196831 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w" pod "2e8113cd-2845-4279-929b-45ef4f8c798b" (UID: "2e8113cd-2845-4279-929b-45ef4f8c798b"). InnerVolumeSpecName "kube-api-access-w454w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:37.295314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.295275 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:37.295314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.295308 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w454w\" (UniqueName: \"kubernetes.io/projected/2e8113cd-2845-4279-929b-45ef4f8c798b-kube-api-access-w454w\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:37.295314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.295322 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e8113cd-2845-4279-929b-45ef4f8c798b-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:37.673666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.673632 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:37.946938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.946850 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" Nov 03 16:34:37.946938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.946897 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj" event={"ID":"2e8113cd-2845-4279-929b-45ef4f8c798b","Type":"ContainerDied","Data":"70c7138e055d3d29144b4812c82dad1198a531fdeb24d1a875c9047006519179"} Nov 03 16:34:37.947182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:37.946970 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c7138e055d3d29144b4812c82dad1198a531fdeb24d1a875c9047006519179" Nov 03 16:34:39.172118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:39.172073 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:40.911819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:40.911785 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:41.480564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.480526 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:41.622285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.622253 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:41.976876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.976793 4539 generic.go:356] "Generic (PLEG): container finished" podID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerID="da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520" exitCode=0 Nov 03 16:34:41.976876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.976856 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" event={"ID":"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c","Type":"ContainerDied","Data":"da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520"} Nov 03 16:34:41.977324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.977208 4539 scope.go:122] "RemoveContainer" containerID="da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520" Nov 03 16:34:41.978322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.978294 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:34:41.978457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.978370 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:34:41.978457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:41.978415 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:34:42.130505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:42.130467 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf"] Nov 03 16:34:42.278873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:42.278841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:42.425794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:42.425764 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:42.568499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:42.568361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:42.868163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:42.868085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:42.989607 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:42.989573 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" event={"ID":"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c","Type":"ContainerStarted","Data":"048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96"} Nov 03 16:34:43.032468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.032410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.191349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.184324 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:43.191349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.185193 4539 patch_prober.go:29] interesting pod/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 16:34:43.191349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.185273 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 16:34:43.191349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.187749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.338274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.338201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.482795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.482711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.638698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.638656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.786376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.786340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.938788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.938743 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj"] Nov 03 16:34:43.940462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.940417 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:43.946571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:43.946542 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fa1268-d2qnj"] Nov 03 16:34:44.116269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.116081 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:44.171240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.171204 4539 patch_prober.go:29] interesting pod/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 16:34:44.171456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.171268 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 16:34:44.262067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.262029 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:44.431511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.431038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:44.468851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.468824 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8113cd-2845-4279-929b-45ef4f8c798b" path="/var/lib/kubelet/pods/2e8113cd-2845-4279-929b-45ef4f8c798b/volumes" Nov 03 16:34:44.612449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.607320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:44.762864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.760451 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:44.939369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:44.939330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:45.129802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:45.129764 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:45.171084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:45.171046 4539 patch_prober.go:29] interesting pod/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 16:34:45.171288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:45.171100 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 16:34:45.395843 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:45.395738 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:45.548468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:45.548367 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:45.998454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:45.998393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:46.171067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:46.171039 4539 patch_prober.go:29] interesting pod/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 16:34:46.171486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:46.171088 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 16:34:46.207450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:46.204832 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:46.420958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:46.420924 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:46.583477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:46.579229 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:47.131566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.131515 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" podUID="c1882612-d579-48ef-9ad8-d7018e81b1f3" containerName="payload-pre-processing" containerID="cri-o://1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c" gracePeriod=25 Nov 03 16:34:47.170963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.170933 4539 patch_prober.go:29] interesting pod/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 16:34:47.171069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.170989 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 16:34:47.462782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.462755 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:47.561916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.561876 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume\") pod \"c1882612-d579-48ef-9ad8-d7018e81b1f3\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " Nov 03 16:34:47.562091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.562016 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm\" (UniqueName: \"kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm\") pod \"c1882612-d579-48ef-9ad8-d7018e81b1f3\" (UID: \"c1882612-d579-48ef-9ad8-d7018e81b1f3\") " Nov 03 16:34:47.562276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.562249 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume" pod "c1882612-d579-48ef-9ad8-d7018e81b1f3" (UID: "c1882612-d579-48ef-9ad8-d7018e81b1f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:47.562533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.562515 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1882612-d579-48ef-9ad8-d7018e81b1f3-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:47.567219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.567187 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm" pod "c1882612-d579-48ef-9ad8-d7018e81b1f3" (UID: "c1882612-d579-48ef-9ad8-d7018e81b1f3"). InnerVolumeSpecName "kube-api-access-7wtfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:47.663227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.663192 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtfm\" (UniqueName: \"kubernetes.io/projected/c1882612-d579-48ef-9ad8-d7018e81b1f3-kube-api-access-7wtfm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:47.790669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:47.790640 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:48.051714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.051627 4539 generic.go:356] "Generic (PLEG): container finished" podID="c1882612-d579-48ef-9ad8-d7018e81b1f3" containerID="1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c" exitCode=0 Nov 03 16:34:48.051714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.051670 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" event={"ID":"c1882612-d579-48ef-9ad8-d7018e81b1f3","Type":"ContainerDied","Data":"1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c"} Nov 03 16:34:48.051940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.051709 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" Nov 03 16:34:48.051940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.051726 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf" event={"ID":"c1882612-d579-48ef-9ad8-d7018e81b1f3","Type":"ContainerDied","Data":"ede4e262f3e20bcf944ed1e14dd9801e79ba39e6d6ddccb3b06643e37eacb7f6"} Nov 03 16:34:48.051940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.051754 4539 scope.go:122] "RemoveContainer" containerID="1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c" Nov 03 16:34:48.061738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.061717 4539 scope.go:122] "RemoveContainer" containerID="1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c" Nov 03 16:34:48.062054 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:34:48.062027 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c\": container with ID starting with 1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c not found: ID does not exist" containerID="1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c" Nov 03 16:34:48.062122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.062066 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c"} err="failed to get container status \"1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c\": rpc error: code = NotFound desc = could not find container \"1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c\": container with ID starting with 1345010a07c79d0de884da91aaa7cf9fd376296391c813238613341cdd9e175c not found: ID does not exist" Nov 03 16:34:48.097412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.097381 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf"] Nov 03 16:34:48.108219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.108189 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa1268-6868f59667-zfrbf"] Nov 03 16:34:48.158681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.158650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:48.191986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.191959 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:48.195742 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.195717 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:34:48.307682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.307569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:48.467633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.467605 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1882612-d579-48ef-9ad8-d7018e81b1f3" path="/var/lib/kubelet/pods/c1882612-d579-48ef-9ad8-d7018e81b1f3/volumes" Nov 03 16:34:48.899235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:48.899206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:49.049549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:49.049520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:49.235389 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:49.235318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:49.399450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:49.399392 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:49.856148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:49.856111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:50.087556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:50.087481 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:50.323731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:50.323695 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:50.475362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:50.475317 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:50.707705 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:50.707613 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:50.875306 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:50.875264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:51.080010 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:51.079965 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:51.554496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:51.554466 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:52.158761 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:52.158713 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:52.313620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:52.313585 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:52.476623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:52.476551 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:52.623541 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:52.623512 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:53.444137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:53.444103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:53.714390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:53.714310 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:53.953059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:53.953024 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms"] Nov 03 16:34:53.987653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:53.987579 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f"] Nov 03 16:34:54.188044 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:54.188015 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:54.369357 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:54.369322 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:54.532557 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:54.532530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:54.696641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:54.696547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:55.842562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:55.842509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:56.009473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:56.009413 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:56.197163 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:34:56.197070 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:34:56.770260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:56.770215 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:56.930064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:56.930019 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:57.083346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:57.083307 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:57.272057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:57.272023 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:57.306197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:57.306161 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh"] Nov 03 16:34:57.306601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:57.306562 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" containerID="cri-o://048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96" gracePeriod=30 Nov 03 16:34:57.424255 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:57.424146 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:57.895900 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:57.895853 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.054133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.054091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.228005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.227934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.680000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.679968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.838295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.838264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.954325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.954239 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" podUID="0610cd84-1893-46b0-ab15-5186095b8d39" containerName="payload-processing" containerID="cri-o://27e71c420275cdc2aee43fe045d6a7e10addbf84947405723b4dfcd8a7432b28" gracePeriod=25 Nov 03 16:34:58.980800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.980773 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:34:58.988511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:58.988480 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" podUID="02398b84-8f13-4118-95e4-87cd66678884" containerName="payload-pre-processing" containerID="cri-o://ffccd929327f49f7f44d3285211896fdb1a680317331ab93edf6bcd2a6d0662b" gracePeriod=25 Nov 03 16:34:59.124478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.124441 4539 generic.go:356] "Generic (PLEG): container finished" podID="0610cd84-1893-46b0-ab15-5186095b8d39" containerID="27e71c420275cdc2aee43fe045d6a7e10addbf84947405723b4dfcd8a7432b28" exitCode=0 Nov 03 16:34:59.124906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.124548 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" event={"ID":"0610cd84-1893-46b0-ab15-5186095b8d39","Type":"ContainerDied","Data":"27e71c420275cdc2aee43fe045d6a7e10addbf84947405723b4dfcd8a7432b28"} Nov 03 16:34:59.128729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.126554 4539 generic.go:356] "Generic (PLEG): container finished" podID="02398b84-8f13-4118-95e4-87cd66678884" containerID="ffccd929327f49f7f44d3285211896fdb1a680317331ab93edf6bcd2a6d0662b" exitCode=0 Nov 03 16:34:59.128729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.126632 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" event={"ID":"02398b84-8f13-4118-95e4-87cd66678884","Type":"ContainerDied","Data":"ffccd929327f49f7f44d3285211896fdb1a680317331ab93edf6bcd2a6d0662b"} Nov 03 16:34:59.244005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.243974 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:34:59.270941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.270918 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:34:59.280895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.280871 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume\") pod \"0610cd84-1893-46b0-ab15-5186095b8d39\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " Nov 03 16:34:59.281022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.280960 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68\" (UniqueName: \"kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68\") pod \"0610cd84-1893-46b0-ab15-5186095b8d39\" (UID: \"0610cd84-1893-46b0-ab15-5186095b8d39\") " Nov 03 16:34:59.281304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.281277 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume" pod "0610cd84-1893-46b0-ab15-5186095b8d39" (UID: "0610cd84-1893-46b0-ab15-5186095b8d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:59.283112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.283092 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68" pod "0610cd84-1893-46b0-ab15-5186095b8d39" (UID: "0610cd84-1893-46b0-ab15-5186095b8d39"). InnerVolumeSpecName "kube-api-access-4wj68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:59.382298 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.382268 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume\") pod \"02398b84-8f13-4118-95e4-87cd66678884\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " Nov 03 16:34:59.382494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.382369 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb\" (UniqueName: \"kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb\") pod \"02398b84-8f13-4118-95e4-87cd66678884\" (UID: \"02398b84-8f13-4118-95e4-87cd66678884\") " Nov 03 16:34:59.382659 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.382623 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume" pod "02398b84-8f13-4118-95e4-87cd66678884" (UID: "02398b84-8f13-4118-95e4-87cd66678884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:59.382757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.382657 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0610cd84-1893-46b0-ab15-5186095b8d39-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:59.382757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.382677 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wj68\" (UniqueName: \"kubernetes.io/projected/0610cd84-1893-46b0-ab15-5186095b8d39-kube-api-access-4wj68\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:59.384532 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.384508 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb" pod "02398b84-8f13-4118-95e4-87cd66678884" (UID: "02398b84-8f13-4118-95e4-87cd66678884"). InnerVolumeSpecName "kube-api-access-5dttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:59.483416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.483334 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dttb\" (UniqueName: \"kubernetes.io/projected/02398b84-8f13-4118-95e4-87cd66678884-kube-api-access-5dttb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:34:59.483416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:34:59.483361 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02398b84-8f13-4118-95e4-87cd66678884-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:00.139456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.139412 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" Nov 03 16:35:00.139935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.139448 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms" event={"ID":"0610cd84-1893-46b0-ab15-5186095b8d39","Type":"ContainerDied","Data":"d724ccf08fa7669238429ce59b4057670814beed8083e4f77aa0956b9315a41d"} Nov 03 16:35:00.139935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.139540 4539 scope.go:122] "RemoveContainer" containerID="27e71c420275cdc2aee43fe045d6a7e10addbf84947405723b4dfcd8a7432b28" Nov 03 16:35:00.141007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.140978 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" Nov 03 16:35:00.141157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.141000 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f" event={"ID":"02398b84-8f13-4118-95e4-87cd66678884","Type":"ContainerDied","Data":"4403d998bd7bedea4c2755d0f970f8c3462159c9f71c3f29c4cd322e2f5caec2"} Nov 03 16:35:00.151708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.151687 4539 scope.go:122] "RemoveContainer" containerID="ffccd929327f49f7f44d3285211896fdb1a680317331ab93edf6bcd2a6d0662b" Nov 03 16:35:00.188785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.188754 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms"] Nov 03 16:35:00.198024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.197998 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5492de-766db9bd9c-zqgms"] Nov 03 16:35:00.229497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.229407 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f"] Nov 03 16:35:00.233149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.233124 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4d5492de-5bfb89b99f-lnh2f"] Nov 03 16:35:00.467256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.467226 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02398b84-8f13-4118-95e4-87cd66678884" path="/var/lib/kubelet/pods/02398b84-8f13-4118-95e4-87cd66678884/volumes" Nov 03 16:35:00.467687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.467667 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0610cd84-1893-46b0-ab15-5186095b8d39" path="/var/lib/kubelet/pods/0610cd84-1893-46b0-ab15-5186095b8d39/volumes" Nov 03 16:35:00.692164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:00.692123 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:02.314278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.314246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:02.473173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.473148 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:02.682361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.682335 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:35:02.710677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710645 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.710828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710694 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.710828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710722 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.710828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710775 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.710828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710815 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.711061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710894 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.711061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710939 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.711061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.710974 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.711061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711053 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert\") pod \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\" (UID: \"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c\") " Nov 03 16:35:02.711254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711056 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:35:02.711372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711278 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:35:02.711458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711379 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.711519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711491 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.711904 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711615 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:35:02.712338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.712319 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:35:02.712499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.711477 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:35:02.714895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.714866 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:35:02.714895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.714869 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:35:02.715077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.714901 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "kube-api-access-zjrpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:35:02.715077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.714963 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy" pod "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" (UID: "87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:35:02.812865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812829 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.812865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812858 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.812865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812867 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.812865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812876 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.813197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812885 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjrpw\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-kube-api-access-zjrpw\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.813197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812894 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.813197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.812902 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:35:02.964887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.964813 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98"] Nov 03 16:35:02.965184 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965173 4539 cpu_manager.go:414] "containerMap: removing container" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" Nov 03 16:35:02.965226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965186 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" Nov 03 16:35:02.965226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965197 4539 cpu_manager.go:414] "containerMap: removing container" podUID="2e8113cd-2845-4279-929b-45ef4f8c798b" containerName="revoke-keys" Nov 03 16:35:02.965226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965202 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8113cd-2845-4279-929b-45ef4f8c798b" containerName="revoke-keys" Nov 03 16:35:02.965226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965218 4539 cpu_manager.go:414] "containerMap: removing container" podUID="02398b84-8f13-4118-95e4-87cd66678884" containerName="payload-pre-processing" Nov 03 16:35:02.965226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965223 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02398b84-8f13-4118-95e4-87cd66678884" containerName="payload-pre-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965233 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0610cd84-1893-46b0-ab15-5186095b8d39" containerName="payload-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965237 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0610cd84-1893-46b0-ab15-5186095b8d39" containerName="payload-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965255 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c1882612-d579-48ef-9ad8-d7018e81b1f3" containerName="payload-pre-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965259 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1882612-d579-48ef-9ad8-d7018e81b1f3" containerName="payload-pre-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965317 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="02398b84-8f13-4118-95e4-87cd66678884" containerName="payload-pre-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965325 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0610cd84-1893-46b0-ab15-5186095b8d39" containerName="payload-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965332 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1882612-d579-48ef-9ad8-d7018e81b1f3" containerName="payload-pre-processing" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965339 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" Nov 03 16:35:02.965378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.965344 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8113cd-2845-4279-929b-45ef4f8c798b" containerName="revoke-keys" Nov 03 16:35:02.968754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.968735 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:02.983668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.983647 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-71d7b4-serving-cert\"" Nov 03 16:35:02.983772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.983672 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 16:35:02.983772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.983703 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7vlf\"" Nov 03 16:35:02.999534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:02.999499 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98"] Nov 03 16:35:03.014951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.014920 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.015144 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.015119 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ws8\" (UniqueName: \"kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.015220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.015185 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.015303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.015287 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.109835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.109802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:03.116271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.116243 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ws8\" (UniqueName: \"kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.116363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.116304 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.116363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.116351 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.116479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.116378 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.116539 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:35:03.116495 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-serving-cert: secret "maas-api-e2e-worker-w0-71d7b4-serving-cert" not found Nov 03 16:35:03.116608 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:35:03.116595 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls podName:ccb45b49-1a1d-4825-beb5-4cf8f983ce97 nodeName:}" failed. No retries permitted until 2026-11-03 16:35:03.616574228 +0000 UTC m=+1247.309845245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls") pod "maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" (UID: "ccb45b49-1a1d-4825-beb5-4cf8f983ce97") : secret "maas-api-e2e-worker-w0-71d7b4-serving-cert" not found Nov 03 16:35:03.117039 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.117023 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.118822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.118801 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.140866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.140835 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ws8\" (UniqueName: \"kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.162089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.162063 4539 generic.go:356] "Generic (PLEG): container finished" podID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerID="048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96" exitCode=0 Nov 03 16:35:03.162212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.162106 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" Nov 03 16:35:03.162212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.162136 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" event={"ID":"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c","Type":"ContainerDied","Data":"048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96"} Nov 03 16:35:03.162212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.162182 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh" event={"ID":"87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c","Type":"ContainerDied","Data":"e963a4ce2505e08d619135d3847a961957650f3df10281a339e0c293e8a7cc27"} Nov 03 16:35:03.162212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.162206 4539 scope.go:122] "RemoveContainer" containerID="048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96" Nov 03 16:35:03.175501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.175481 4539 scope.go:122] "RemoveContainer" containerID="da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520" Nov 03 16:35:03.184281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.184253 4539 scope.go:122] "RemoveContainer" containerID="048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96" Nov 03 16:35:03.184569 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:35:03.184552 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96\": container with ID starting with 048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96 not found: ID does not exist" containerID="048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96" Nov 03 16:35:03.184630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.184580 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96"} err="failed to get container status \"048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96\": rpc error: code = NotFound desc = could not find container \"048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96\": container with ID starting with 048202e0cd7329ede791476a59e85a0dcd61bcc0fa2ce3a4a8cf9460e4cd4d96 not found: ID does not exist" Nov 03 16:35:03.184630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.184601 4539 scope.go:122] "RemoveContainer" containerID="da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520" Nov 03 16:35:03.184860 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:35:03.184842 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520\": container with ID starting with da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520 not found: ID does not exist" containerID="da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520" Nov 03 16:35:03.184907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.184867 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520"} err="failed to get container status \"da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520\": rpc error: code = NotFound desc = could not find container \"da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520\": container with ID starting with da636fe8d2af73628a1d6aa819e8699d9186d957d84b47877ed5fecc5aa66520 not found: ID does not exist" Nov 03 16:35:03.214020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.213993 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh"] Nov 03 16:35:03.217744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.217674 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4d5492de-openshift-default-6c5c74f5fc-kcjlh"] Nov 03 16:35:03.266551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.266515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:03.411473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.411418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:03.621292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.621258 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.624035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.624007 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\") pod \"maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:03.662359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.662334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:03.881255 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:03.881167 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:04.199538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:04.199514 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98"] Nov 03 16:35:04.201048 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:35:04.201022 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb45b49_1a1d_4825_beb5_4cf8f983ce97.slice/crio-2a56463e45c77c051e80d5c437da236cad398017d2974f6c863a6d1eb9996cc4 WatchSource:0}: Error finding container 2a56463e45c77c051e80d5c437da236cad398017d2974f6c863a6d1eb9996cc4: Status 404 returned error can't find the container with id 2a56463e45c77c051e80d5c437da236cad398017d2974f6c863a6d1eb9996cc4 Nov 03 16:35:04.469452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:04.469347 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" path="/var/lib/kubelet/pods/87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c/volumes" Nov 03 16:35:04.488691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:04.488654 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:04.671763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:04.671736 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:05.180955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:05.180879 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" event={"ID":"ccb45b49-1a1d-4825-beb5-4cf8f983ce97","Type":"ContainerStarted","Data":"2a56463e45c77c051e80d5c437da236cad398017d2974f6c863a6d1eb9996cc4"} Nov 03 16:35:05.328190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:05.327673 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:06.243151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:06.243076 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:06.396685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:06.396651 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:07.195128 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.195094 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" event={"ID":"ccb45b49-1a1d-4825-beb5-4cf8f983ce97","Type":"ContainerStarted","Data":"d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142"} Nov 03 16:35:07.195522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.195264 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:07.244933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.244881 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" podStartSLOduration=2.344946965 podStartE2EDuration="5.244863576s" totalImagesPullingTime="2.899916611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:07.243587032 +0000 UTC m=+1250.936858082" watchObservedRunningTime="2026-11-03 16:35:07.244863576 +0000 UTC m=+1250.938134598" Nov 03 16:35:07.334501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.334471 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:07.582213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.582180 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c"] Nov 03 16:35:07.582608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.582590 4539 cpu_manager.go:414] "containerMap: removing container" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" Nov 03 16:35:07.582608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.582606 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" Nov 03 16:35:07.582773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.582702 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c2e2ab-8e54-4c80-b760-7ff8c98f6f9c" containerName="istio-proxy" Nov 03 16:35:07.585932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.585911 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.599121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.599088 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"kube-root-ca.crt\"" Nov 03 16:35:07.599250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.599215 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"default-dockercfg-v9hmw\"" Nov 03 16:35:07.609455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.609407 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"simulator-w2-7519b7-kserve-self-signed-certs\"" Nov 03 16:35:07.609583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.609530 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"openshift-service-ca.crt\"" Nov 03 16:35:07.612464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.612444 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c"] Nov 03 16:35:07.664554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.664524 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.664735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.664557 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.664735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.664666 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.664735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.664714 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7zn\" (UniqueName: \"kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.664888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.664736 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.664939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.664916 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.669824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.669801 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:07.766166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766132 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766173 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766191 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766216 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766244 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7zn\" (UniqueName: \"kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766261 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766595 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766678 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766662 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.766782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.766764 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.768674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.768654 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.768914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.768896 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.788353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.788328 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7zn\" (UniqueName: \"kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn\") pod \"simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:07.911999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:07.911913 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:08.213792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:08.213745 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c"] Nov 03 16:35:08.215188 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:35:08.215160 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb525d2d_1f9c_4354_90be_18439f5ad7a4.slice/crio-ac5e7daff2590d88e7c431e6b5d6e37c6c6d1c875386d42503dfcbbe36e9c83e WatchSource:0}: Error finding container ac5e7daff2590d88e7c431e6b5d6e37c6c6d1c875386d42503dfcbbe36e9c83e: Status 404 returned error can't find the container with id ac5e7daff2590d88e7c431e6b5d6e37c6c6d1c875386d42503dfcbbe36e9c83e Nov 03 16:35:08.861039 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:08.861002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:09.214907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:09.214821 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" event={"ID":"cb525d2d-1f9c-4354-90be-18439f5ad7a4","Type":"ContainerStarted","Data":"ac5e7daff2590d88e7c431e6b5d6e37c6c6d1c875386d42503dfcbbe36e9c83e"} Nov 03 16:35:09.319076 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:09.319030 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:09.608345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:09.608314 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:09.765403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:09.765346 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:10.226199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:10.226165 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" event={"ID":"cb525d2d-1f9c-4354-90be-18439f5ad7a4","Type":"ContainerStarted","Data":"867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e"} Nov 03 16:35:10.275533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:10.275470 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podStartSLOduration=1.3561368360000001 podStartE2EDuration="3.275454337s" totalImagesPullingTime="1.919317501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:10.272378499 +0000 UTC m=+1253.965649520" watchObservedRunningTime="2026-11-03 16:35:10.275454337 +0000 UTC m=+1253.968725359" Nov 03 16:35:10.912201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:10.912161 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:11.303473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:11.303411 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:13.101898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:13.101858 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:13.206363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:13.206337 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 16:35:13.252864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:13.252825 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:15.302545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:15.302501 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:15.464417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:15.464385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:15.619252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:15.619141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:16.200686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:16.200648 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:17.653839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:17.653816 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:17.912918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:17.912801 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:17.928466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:17.928408 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:18.284881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:18.284854 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:35:19.458781 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:19.458747 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:19.986239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:19.986194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:20.137273 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:20.137235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:20.282443 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:20.282394 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:21.402832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:21.402797 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:22.978907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:22.978874 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:23.299140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.299105 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs"] Nov 03 16:35:23.308018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.307979 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.327456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.327401 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs"] Nov 03 16:35:23.334386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.334349 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"simulator-w4-460104-kserve-self-signed-certs\"" Nov 03 16:35:23.334386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.334379 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"openshift-service-ca.crt\"" Nov 03 16:35:23.334608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.334388 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"kube-root-ca.crt\"" Nov 03 16:35:23.334608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.334449 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"default-dockercfg-2xrq7\"" Nov 03 16:35:23.383383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.383341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:23.432999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.432962 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.433194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.433031 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.433194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.433111 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.433305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.433192 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.433305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.433222 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.433371 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.433316 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sht4\" (UniqueName: \"kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.534687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.534651 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.534898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.534715 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.534898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.534738 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.534898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.534760 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.534898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.534780 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.534898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.534809 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sht4\" (UniqueName: \"kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.535799 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.535737 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.536328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.536305 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.536619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.536570 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.540222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.539647 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.541260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.541234 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.556885 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.556827 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sht4\" (UniqueName: \"kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4\") pod \"simulator-w4-460104-kserve-5756956476-gsdgs\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.622393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.622358 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:23.931565 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:23.931540 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs"] Nov 03 16:35:23.933393 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:35:23.933370 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e552f18_887c_49cf_88c4_f043aa0d5ddf.slice/crio-d03df31a1c3b144158ede92e55e4d4269046c60c679dd9f17f72b61407fcde70 WatchSource:0}: Error finding container d03df31a1c3b144158ede92e55e4d4269046c60c679dd9f17f72b61407fcde70: Status 404 returned error can't find the container with id d03df31a1c3b144158ede92e55e4d4269046c60c679dd9f17f72b61407fcde70 Nov 03 16:35:24.308642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:24.308604 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" event={"ID":"0e552f18-887c-49cf-88c4-f043aa0d5ddf","Type":"ContainerStarted","Data":"443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba"} Nov 03 16:35:24.308642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:24.308648 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" event={"ID":"0e552f18-887c-49cf-88c4-f043aa0d5ddf","Type":"ContainerStarted","Data":"d03df31a1c3b144158ede92e55e4d4269046c60c679dd9f17f72b61407fcde70"} Nov 03 16:35:24.349274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:24.349203 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podStartSLOduration=1.34918513 podStartE2EDuration="1.34918513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:24.347572899 +0000 UTC m=+1268.040843932" watchObservedRunningTime="2026-11-03 16:35:24.34918513 +0000 UTC m=+1268.042456149" Nov 03 16:35:24.597545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:24.597475 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:24.622532 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:24.622496 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:26.156716 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:35:26.156673 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:35:26.205732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:26.205701 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:28.027482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:28.027449 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:28.241387 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:28.241344 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:28.392310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:28.392234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:29.416069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:29.416040 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:29.892771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:29.892733 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:31.569838 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:31.569803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:33.248885 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.248854 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:33.305443 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.305400 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp"] Nov 03 16:35:33.310210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.310187 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.333142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.333116 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"premium-w2-7519b7-kserve-self-signed-certs\"" Nov 03 16:35:33.336375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.336350 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp"] Nov 03 16:35:33.407121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.407087 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:33.431329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.431294 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.431329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.431328 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.431604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.431350 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.431604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.431515 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.431604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.431543 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp85x\" (UniqueName: \"kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.431771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.431652 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533103 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533153 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533181 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533230 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533255 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp85x\" (UniqueName: \"kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533355 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533677 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.533962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533935 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.534142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.533978 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.536129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.536104 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.536363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.536344 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.556135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.556099 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp85x\" (UniqueName: \"kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x\") pod \"premium-w2-7519b7-kserve-55bdb789d7-mqjsp\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.622869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.622836 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:33.623062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.622899 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:33.637303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.637275 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:33.923478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:33.923451 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp"] Nov 03 16:35:33.926180 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:35:33.926155 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef6de8c8_3034_4a43_8e58_04dab26ee9de.slice/crio-6ae935189a183808350d4bf929ac08e1d41ce2da798e956dbeb2947b78ab641f WatchSource:0}: Error finding container 6ae935189a183808350d4bf929ac08e1d41ce2da798e956dbeb2947b78ab641f: Status 404 returned error can't find the container with id 6ae935189a183808350d4bf929ac08e1d41ce2da798e956dbeb2947b78ab641f Nov 03 16:35:34.377372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:34.377331 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" event={"ID":"ef6de8c8-3034-4a43-8e58-04dab26ee9de","Type":"ContainerStarted","Data":"22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402"} Nov 03 16:35:34.377909 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:34.377388 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" event={"ID":"ef6de8c8-3034-4a43-8e58-04dab26ee9de","Type":"ContainerStarted","Data":"6ae935189a183808350d4bf929ac08e1d41ce2da798e956dbeb2947b78ab641f"} Nov 03 16:35:34.388739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:34.388716 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:35:34.418257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:34.418121 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podStartSLOduration=1.418101437 podStartE2EDuration="1.418101437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:34.416186512 +0000 UTC m=+1278.109457534" watchObservedRunningTime="2026-11-03 16:35:34.418101437 +0000 UTC m=+1278.111372461" Nov 03 16:35:34.623279 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:34.623246 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:34.801778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:34.801728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:36.408073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:36.408039 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:38.183926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:38.183887 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:39.247723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:39.247686 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:40.018418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:40.018380 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:41.572838 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:41.572803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:43.187459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:43.187405 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:43.623439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:43.623379 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:43.636391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:43.636360 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:44.459379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:44.459345 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:35:44.837508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:44.837469 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:46.561757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:46.561722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:47.874573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:47.874530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:48.162339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:48.162259 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:49.883317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:49.883283 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:51.580955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:51.580926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:53.277983 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:53.277943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:54.547766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:54.547721 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:54.994160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:54.994074 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:56.156789 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:35:56.156749 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:35:56.760063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:56.760015 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:35:58.411844 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:35:58.411812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:00.051752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.051715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:00.790348 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.790311 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw"] Nov 03 16:36:00.799127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.799097 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.822745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.822717 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"default-dockercfg-h4hqv\"" Nov 03 16:36:00.822745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.822717 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"openshift-service-ca.crt\"" Nov 03 16:36:00.823037 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.823017 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"premium-w0-71d7b4-kserve-self-signed-certs\"" Nov 03 16:36:00.823095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.823049 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"kube-root-ca.crt\"" Nov 03 16:36:00.827114 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.827092 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw"] Nov 03 16:36:00.888703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.888661 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.888703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.888699 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcj7b\" (UniqueName: \"kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.888963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.888724 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.888963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.888859 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.888963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.888891 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.888963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.888908 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.903440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.903382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:00.991469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.991153 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.991469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.991216 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.991469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.991246 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.991469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.991346 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.991469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.991386 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcj7b\" (UniqueName: \"kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.991469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.991458 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.995450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.992283 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.995450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.992874 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.995450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.993316 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.997183 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.996196 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:00.997183 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:00.996857 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:01.014416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.014380 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcj7b\" (UniqueName: \"kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b\") pod \"premium-w0-71d7b4-kserve-5995b86d95-gqmkw\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:01.114492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.114383 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:01.424554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.424505 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw"] Nov 03 16:36:01.427075 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:36:01.427047 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2469139_15c6_45eb_ad72_ab7f4ef38e4e.slice/crio-4fee29f13d23b8184727ea3982f534d80c2d8f0a5b6831d91ff3786e06e726d5 WatchSource:0}: Error finding container 4fee29f13d23b8184727ea3982f534d80c2d8f0a5b6831d91ff3786e06e726d5: Status 404 returned error can't find the container with id 4fee29f13d23b8184727ea3982f534d80c2d8f0a5b6831d91ff3786e06e726d5 Nov 03 16:36:01.545807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.545772 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" event={"ID":"c2469139-15c6-45eb-ad72-ab7f4ef38e4e","Type":"ContainerStarted","Data":"b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d"} Nov 03 16:36:01.545807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.545821 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" event={"ID":"c2469139-15c6-45eb-ad72-ab7f4ef38e4e","Type":"ContainerStarted","Data":"4fee29f13d23b8184727ea3982f534d80c2d8f0a5b6831d91ff3786e06e726d5"} Nov 03 16:36:01.587822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.587762 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podStartSLOduration=1.5877439309999999 podStartE2EDuration="1.587743931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:01.585873899 +0000 UTC m=+1305.279144928" watchObservedRunningTime="2026-11-03 16:36:01.587743931 +0000 UTC m=+1305.281014954" Nov 03 16:36:01.805300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:01.805264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:02.114830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:02.114735 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:03.380028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:03.379998 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:05.182715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:05.182687 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:06.852686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:06.852645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:08.609986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:08.609951 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:08.977034 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:08.976949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:10.114202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:10.114170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:11.115401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:11.115366 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:11.128718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:11.128687 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:11.611976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:11.611942 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:36:11.668631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:11.668596 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:13.270031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:13.269988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:14.842315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.842275 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4"] Nov 03 16:36:14.846780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.846760 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.871663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.871638 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3a2fa2-openshift-default-dockercfg-q9n76\"" Nov 03 16:36:14.892325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.892299 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4"] Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933104 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933199 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933251 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933280 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933305 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933351 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933443 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933473 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.933874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.933500 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnh5q\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:14.968256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:14.968189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:15.034614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034582 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034618 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034644 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnh5q\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034711 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034771 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034812 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034842 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.034865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034865 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.035137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.034985 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.035137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.035035 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.035472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.035411 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.035604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.035511 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.035675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.035537 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.035675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.035664 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.037559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.037538 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.038649 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.038626 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.056794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.056759 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.056932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.056826 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnh5q\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.160047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.159961 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:15.503521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.503395 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4"] Nov 03 16:36:15.507311 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:36:15.507266 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413cb794_f9b6_4a79_913d_6f0c14921c86.slice/crio-8870e550f25a5564bcb439252957d00bcd57b2e7d199e66d8707d0a06d07a216 WatchSource:0}: Error finding container 8870e550f25a5564bcb439252957d00bcd57b2e7d199e66d8707d0a06d07a216: Status 404 returned error can't find the container with id 8870e550f25a5564bcb439252957d00bcd57b2e7d199e66d8707d0a06d07a216 Nov 03 16:36:15.509943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.509909 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:15.510065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.509992 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:15.510065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.510040 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:15.544911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.544882 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:15.626331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.626259 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs"] Nov 03 16:36:15.696521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.695466 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs"] Nov 03 16:36:15.696521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.695534 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" event={"ID":"413cb794-f9b6-4a79-913d-6f0c14921c86","Type":"ContainerStarted","Data":"7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca"} Nov 03 16:36:15.696521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.695568 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" event={"ID":"413cb794-f9b6-4a79-913d-6f0c14921c86","Type":"ContainerStarted","Data":"8870e550f25a5564bcb439252957d00bcd57b2e7d199e66d8707d0a06d07a216"} Nov 03 16:36:15.696521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.695675 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.710670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.710622 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7519b7-openshift-default-dockercfg-dv7k2\"" Nov 03 16:36:15.748784 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.748716 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podStartSLOduration=1.7486968840000001 podStartE2EDuration="1.748696884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:15.745782673 +0000 UTC m=+1319.439053695" watchObservedRunningTime="2026-11-03 16:36:15.748696884 +0000 UTC m=+1319.441967898" Nov 03 16:36:15.853697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.853660 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.853724 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsrv\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.853949 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.853993 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.854028 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.854160 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.854210 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.854252 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.854417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.854338 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.955867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.955833 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.955867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.955871 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.955888 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.955909 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.955931 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.955951 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956094 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956207 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956256 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsrv\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956372 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956514 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956605 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956548 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956718 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.956999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.956979 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.958862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.958839 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.959201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.959177 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.980321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.980287 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsrv\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:15.986444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:15.986399 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:16.028689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.028657 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:16.161003 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.160915 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:16.162121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.162099 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:16.162224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.162144 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:16.365065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.365036 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs"] Nov 03 16:36:16.366440 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:36:16.366390 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3743f418_0f1d_46b4_b0db_8f7a50bf4942.slice/crio-e44d8c1b680b1bf11a1cf17bcea1da918b58394de0f7113244beeb030c125feb WatchSource:0}: Error finding container e44d8c1b680b1bf11a1cf17bcea1da918b58394de0f7113244beeb030c125feb: Status 404 returned error can't find the container with id e44d8c1b680b1bf11a1cf17bcea1da918b58394de0f7113244beeb030c125feb Nov 03 16:36:16.368877 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.368847 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:16.368966 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.368909 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:16.368966 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.368941 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:16.639449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.638048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:16.719509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.719053 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" event={"ID":"3743f418-0f1d-46b4-b0db-8f7a50bf4942","Type":"ContainerStarted","Data":"8e43281e796d196e520beafc386170056ced47ff95dbe5a2d66a7fadc2a37062"} Nov 03 16:36:16.719509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.719114 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" event={"ID":"3743f418-0f1d-46b4-b0db-8f7a50bf4942","Type":"ContainerStarted","Data":"e44d8c1b680b1bf11a1cf17bcea1da918b58394de0f7113244beeb030c125feb"} Nov 03 16:36:16.769813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:16.769749 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podStartSLOduration=1.769729085 podStartE2EDuration="1.769729085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:16.767707849 +0000 UTC m=+1320.460978890" watchObservedRunningTime="2026-11-03 16:36:16.769729085 +0000 UTC m=+1320.463000110" Nov 03 16:36:17.032450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:17.029795 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:17.036444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:17.034558 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:17.036444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:17.034624 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:17.161012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:17.160969 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:17.161196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:17.161037 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:18.033450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:18.032844 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:18.033450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:18.032905 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:18.160403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:18.160371 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:18.160616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:18.160463 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:18.271172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:18.271139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:19.040450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:19.039817 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:19.040450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:19.039881 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:19.160409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:19.160371 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:19.160639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:19.160443 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:19.865455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:19.865373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:20.032843 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:20.032805 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:20.033146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:20.033119 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:20.167254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:20.167157 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:20.167923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:20.167854 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:21.031450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:21.029656 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:21.031450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:21.029726 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:21.162185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:21.161786 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:21.162185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:21.161840 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:21.488492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:21.488373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:22.030818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:22.030783 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:22.031016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:22.030840 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:22.160990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:22.160951 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 03 16:36:22.161181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:22.161019 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 03 16:36:23.029635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:23.029596 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:36:23.030166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:23.029677 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:36:23.089786 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:23.089740 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:23.165304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:23.165274 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:23.165677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:23.165654 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:23.166302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:23.166284 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 16:36:24.034316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:24.034249 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:24.034844 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:24.034741 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:24.035679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:24.035659 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:36:24.783758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:24.783717 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:26.158299 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:36:26.158259 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:36:26.685155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:26.685099 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:26.840639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:26.840598 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:28.358227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:28.358189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:30.081905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:30.081866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:31.676623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:31.676561 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:33.265122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:33.265091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:34.785900 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:34.785861 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:35.599288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.599252 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg"] Nov 03 16:36:35.602946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.602922 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.616000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.615977 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-71d7b4-openshift-default-dockercfg-5vhxc\"" Nov 03 16:36:35.634622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.634597 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg"] Nov 03 16:36:35.701439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701401 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701453 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701535 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701612 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8gj\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701678 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701733 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701807 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701826 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.701891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.701849 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803183 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8gj\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803219 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803346 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803391 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803418 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803477 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803548 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803575 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803624 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.803825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803653 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.804231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803850 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.804231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.803858 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.804231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.804116 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.804388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.804251 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.805941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.805919 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.806067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.806048 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.824872 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.824843 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.825017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.824982 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8gj\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:35.918391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:35.918297 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:36.242354 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.242326 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg"] Nov 03 16:36:36.243933 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:36:36.243909 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129681f8_da05_4f1f_978b_7998ab2c0c0a.slice/crio-38a762b3fb2dcc9fbe1cbdf15bdc1fe9eb2b95b4994edc72b55d2b68644c8647 WatchSource:0}: Error finding container 38a762b3fb2dcc9fbe1cbdf15bdc1fe9eb2b95b4994edc72b55d2b68644c8647: Status 404 returned error can't find the container with id 38a762b3fb2dcc9fbe1cbdf15bdc1fe9eb2b95b4994edc72b55d2b68644c8647 Nov 03 16:36:36.246543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.246512 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:36.246626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.246579 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:36.246626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.246605 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:36:36.359576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.359539 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:36.882655 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.882620 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" event={"ID":"129681f8-da05-4f1f-978b-7998ab2c0c0a","Type":"ContainerStarted","Data":"c058f0dee5a1fd81c80154cb3eec67f9748a00b6a7fd6ee2bec29983dc7f5e5d"} Nov 03 16:36:36.883061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.882676 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" event={"ID":"129681f8-da05-4f1f-978b-7998ab2c0c0a","Type":"ContainerStarted","Data":"38a762b3fb2dcc9fbe1cbdf15bdc1fe9eb2b95b4994edc72b55d2b68644c8647"} Nov 03 16:36:36.930449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.930021 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:36.938165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:36.938120 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podStartSLOduration=1.938106872 podStartE2EDuration="1.938106872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:36.937937098 +0000 UTC m=+1340.631208122" watchObservedRunningTime="2026-11-03 16:36:36.938106872 +0000 UTC m=+1340.631377894" Nov 03 16:36:37.879216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:37.879183 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:37.941473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:37.940556 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:36:37.941473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:37.940624 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 16:36:38.920449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:38.919826 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:36:38.920449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:38.919889 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 16:36:39.736053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:39.736022 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:39.920941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:39.920906 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:36:39.921271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:39.921223 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 16:36:40.919872 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:40.919350 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:36:40.919872 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:40.919413 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 16:36:41.387066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:41.387033 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:41.918814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:41.918778 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 16:36:41.919109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:41.919078 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 16:36:42.984466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:42.984417 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:42.985510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:42.985474 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:42.986831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:42.986809 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:36:42.989691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:42.989666 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:44.598540 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:44.598507 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:46.367612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:46.367573 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:48.096455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:48.096409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:49.724778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:49.724744 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:51.366028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:51.365987 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:52.987083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:52.987047 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:54.763545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:54.763516 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:56.156008 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:36:56.155976 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:36:56.376318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:56.376268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:58.030526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:58.030488 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:36:59.683396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:36:59.683365 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:01.275657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:01.275621 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:02.986681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:02.986636 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:04.584503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:04.584455 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:06.167052 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:06.167017 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:07.683705 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:07.683667 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:09.091077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:09.091044 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:09.275266 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:09.275234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:10.898112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:10.898072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:11.755241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:11.754500 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:11.907952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:11.907920 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:12.677927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:12.677896 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:12.937500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:12.934236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:13.072547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:13.072515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:13.230671 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:13.230585 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:14.386305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:14.386263 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:15.988765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:15.988728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:17.770143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:17.770111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:19.384377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:19.384308 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:21.088380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:21.088342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:22.987799 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:22.987765 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:24.783260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:24.783227 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:26.157201 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:37:26.157162 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:37:26.369770 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:26.369731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:26.622140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:26.622109 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:26.806939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:26.806908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:28.067307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:28.067278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:29.835451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:29.831472 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:32.053973 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:32.053942 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:33.797987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:33.797927 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:35.476954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:35.476918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:37.139297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:37.139265 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:38.904179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:38.904143 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:40.929190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:40.929153 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:41.630206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:41.630078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:41.798207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:41.798175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:43.311108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:43.311069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:44.298602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:44.298563 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:45.103702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.103670 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q"] Nov 03 16:37:45.107313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.107295 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.124388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.124359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:45.153504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.148563 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q"] Nov 03 16:37:45.210952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.210911 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.210978 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5p4\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211076 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211173 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211205 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211326 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211365 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211379 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.211555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.211398 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312452 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312500 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312533 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312588 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312611 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312635 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312744 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312783 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5p4\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312831 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312862 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.312985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.312920 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.313253 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.313125 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.313491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.313372 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.313625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.313604 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.315086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.315063 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.315798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.315778 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.334765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.334745 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.335114 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.335093 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5p4\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.453530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.453435 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:45.769735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.769711 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q"] Nov 03 16:37:45.772032 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:37:45.772007 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529563fd_3eba_42b4_9f15_bae338d84fd7.slice/crio-20d0dc3f1880813fa2e85ccebd728f8e2ef2da612216cf1a97ea456381083a43 WatchSource:0}: Error finding container 20d0dc3f1880813fa2e85ccebd728f8e2ef2da612216cf1a97ea456381083a43: Status 404 returned error can't find the container with id 20d0dc3f1880813fa2e85ccebd728f8e2ef2da612216cf1a97ea456381083a43 Nov 03 16:37:45.774246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.774213 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:37:45.774315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.774279 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:37:45.774315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:45.774308 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:37:46.381226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:46.381193 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" event={"ID":"529563fd-3eba-42b4-9f15-bae338d84fd7","Type":"ContainerStarted","Data":"ff8aff3444b27de0da89270b300769b3477f4ca661fcd9e78fadc1c8d978132f"} Nov 03 16:37:46.381702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:46.381246 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" event={"ID":"529563fd-3eba-42b4-9f15-bae338d84fd7","Type":"ContainerStarted","Data":"20d0dc3f1880813fa2e85ccebd728f8e2ef2da612216cf1a97ea456381083a43"} Nov 03 16:37:46.453634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:46.453596 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:46.455183 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:46.455160 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:37:46.455321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:46.455214 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 16:37:46.793367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:46.793269 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:47.454565 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:47.454531 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:37:47.454981 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:47.454586 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 16:37:48.453800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:48.453761 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:37:48.454011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:48.453812 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 16:37:48.577996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:48.577925 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:49.340459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:49.340388 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:49.456810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:49.456775 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:37:49.457074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:49.457044 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 16:37:49.499106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:49.499070 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:50.431112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:50.430980 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:50.453830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:50.453790 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:37:50.454029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:50.453883 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 16:37:51.453992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:51.453958 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 16:37:51.454506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:51.454474 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 16:37:51.986617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:51.986580 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:52.500715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:52.500686 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:52.501919 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:52.501894 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:52.503150 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:52.503126 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:37:52.556517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:52.556464 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podStartSLOduration=7.5564493299999995 podStartE2EDuration="7.55644933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:46.431760808 +0000 UTC m=+1410.125031829" watchObservedRunningTime="2026-11-03 16:37:52.55644933 +0000 UTC m=+1416.249720343" Nov 03 16:37:53.585499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:53.585410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:54.673291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:54.673258 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:55.632771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:55.632734 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:56.393447 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:37:56.392711 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:37:57.612449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:57.610321 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:58.173416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.173387 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj"] Nov 03 16:37:58.201903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.196689 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.300472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.291618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:58.300472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.299565 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"e2e-conflict-b-f7f1d684-kserve-self-signed-certs\"" Nov 03 16:37:58.322448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.318678 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj"] Nov 03 16:37:58.451511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.450813 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.475448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.454515 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.475448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.466679 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp86v\" (UniqueName: \"kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.475448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.466756 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.475448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.466782 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.475448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.466816 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.579706 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp86v\" (UniqueName: \"kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.579791 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.579838 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.579870 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.579936 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.579993 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.593770 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.594294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.594024 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.607487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.596886 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.607487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.606024 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp86v\" (UniqueName: \"kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.612930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.607744 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.616444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.616285 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs\") pod \"e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:58.708496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:58.707607 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:37:59.109814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:59.109785 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj"] Nov 03 16:37:59.111451 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:37:59.111406 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22c115d3_9503_4c0e_a3de_bf23aadce1cd.slice/crio-a8f4a7c15b8202bf0d29e5dd574d63268fb1181dd63a98d78efce8e63baa14b1 WatchSource:0}: Error finding container a8f4a7c15b8202bf0d29e5dd574d63268fb1181dd63a98d78efce8e63baa14b1: Status 404 returned error can't find the container with id a8f4a7c15b8202bf0d29e5dd574d63268fb1181dd63a98d78efce8e63baa14b1 Nov 03 16:37:59.296176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:59.296145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:37:59.540382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:59.540336 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" event={"ID":"22c115d3-9503-4c0e-a3de-bf23aadce1cd","Type":"ContainerStarted","Data":"7e57cc6df6381f5d3592b2e2e170a7125f2619e984233361b6c103934e6c841b"} Nov 03 16:37:59.540583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:59.540396 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" event={"ID":"22c115d3-9503-4c0e-a3de-bf23aadce1cd","Type":"ContainerStarted","Data":"a8f4a7c15b8202bf0d29e5dd574d63268fb1181dd63a98d78efce8e63baa14b1"} Nov 03 16:37:59.616110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:59.616050 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" podStartSLOduration=1.616037082 podStartE2EDuration="1.616037082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:59.615554669 +0000 UTC m=+1423.308825693" watchObservedRunningTime="2026-11-03 16:37:59.616037082 +0000 UTC m=+1423.309308109" Nov 03 16:37:59.708459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:37:59.708413 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:00.883661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:00.883610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:01.067459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.067409 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92"] Nov 03 16:38:01.071717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.071690 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.103359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.103323 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92"] Nov 03 16:38:01.111285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111249 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111296 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5jf\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111325 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111375 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111454 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111488 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111580 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111603 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.111717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.111636 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213051 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.212969 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5jf\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213051 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213013 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213051 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213050 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213084 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213106 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213159 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213185 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213219 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213305 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213637 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213646 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213900 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.213925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213920 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.214229 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.213971 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.215859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.215837 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.215957 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.215854 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.234824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.234801 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.235155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.235132 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5jf\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.385489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.385458 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:01.710295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.710273 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92"] Nov 03 16:38:01.713825 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:38:01.713796 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f19163_1b3d_4252_a39e_95d30edfb352.slice/crio-6928fe3cfea132513c787b3e216eafa5b993589bc7dd6af16fe8fcef850e2b41 WatchSource:0}: Error finding container 6928fe3cfea132513c787b3e216eafa5b993589bc7dd6af16fe8fcef850e2b41: Status 404 returned error can't find the container with id 6928fe3cfea132513c787b3e216eafa5b993589bc7dd6af16fe8fcef850e2b41 Nov 03 16:38:01.716011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.715980 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:01.716081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.716047 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:01.716081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:01.716076 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:02.559864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:02.559832 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" event={"ID":"94f19163-1b3d-4252-a39e-95d30edfb352","Type":"ContainerStarted","Data":"60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711"} Nov 03 16:38:02.560309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:02.559890 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" event={"ID":"94f19163-1b3d-4252-a39e-95d30edfb352","Type":"ContainerStarted","Data":"6928fe3cfea132513c787b3e216eafa5b993589bc7dd6af16fe8fcef850e2b41"} Nov 03 16:38:02.626279 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:02.626213 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podStartSLOduration=1.626193094 podStartE2EDuration="1.626193094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:02.622948133 +0000 UTC m=+1426.316219182" watchObservedRunningTime="2026-11-03 16:38:02.626193094 +0000 UTC m=+1426.319464121" Nov 03 16:38:02.721450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:02.712821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:03.386602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:03.386571 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:03.388513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:03.388488 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:38:03.388663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:03.388533 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 16:38:04.386751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:04.386719 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:38:04.387169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:04.386773 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 16:38:04.417800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:04.417767 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:05.386117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:05.386085 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:38:05.386302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:05.386139 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 16:38:06.107036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:06.106906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:06.386832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:06.386756 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:38:06.387046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:06.387025 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 16:38:07.386153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:07.386118 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:38:07.386581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:07.386174 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 16:38:07.894649 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:07.894605 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:08.386602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:08.386558 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 16:38:08.387215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:08.387168 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 16:38:08.716509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:08.694297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:08.717744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:08.716888 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:09.044788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:09.044756 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:09.329395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:09.329302 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:10.097611 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:10.097577 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:10.101447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:10.098170 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:10.108450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:10.106575 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:10.221448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:10.221046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:10.496674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:10.496565 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:10.496967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:10.496914 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:11.480450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:11.476963 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:11.480450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:11.478262 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:11.485820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:11.485792 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 16:38:12.081224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:12.081173 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:13.770170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:13.770136 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:15.423685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:15.423646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:17.315115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:17.315082 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:17.722050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:17.721956 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:19.096933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:19.096892 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:20.804022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:20.803978 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:21.014210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:21.014182 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:21.859774 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:21.858370 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj"] Nov 03 16:38:21.993754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:21.993716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:22.606937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:22.606902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:24.210793 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:24.210756 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:25.934440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:25.934380 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:26.171912 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:38:26.171867 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:38:26.347669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:26.347631 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:27.567316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:27.567287 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:29.307263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:29.307228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:30.209910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:30.209843 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:30.385398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:30.385362 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:31.114158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:31.114122 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:32.679723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:32.679688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:34.379915 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:34.379873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:35.969943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:35.969893 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:36.891452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:36.890605 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerName="main" containerID="cri-o://7e57cc6df6381f5d3592b2e2e170a7125f2619e984233361b6c103934e6c841b" gracePeriod=45 Nov 03 16:38:37.131155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:37.131096 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused" Nov 03 16:38:37.261175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:37.260991 4539 generic.go:356] "Generic (PLEG): container finished" podID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerID="7e57cc6df6381f5d3592b2e2e170a7125f2619e984233361b6c103934e6c841b" exitCode=0 Nov 03 16:38:37.261175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:37.261099 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" event={"ID":"22c115d3-9503-4c0e-a3de-bf23aadce1cd","Type":"ContainerDied","Data":"7e57cc6df6381f5d3592b2e2e170a7125f2619e984233361b6c103934e6c841b"} Nov 03 16:38:38.294448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.293994 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" event={"ID":"22c115d3-9503-4c0e-a3de-bf23aadce1cd","Type":"ContainerDied","Data":"a8f4a7c15b8202bf0d29e5dd574d63268fb1181dd63a98d78efce8e63baa14b1"} Nov 03 16:38:38.294448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.294042 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f4a7c15b8202bf0d29e5dd574d63268fb1181dd63a98d78efce8e63baa14b1" Nov 03 16:38:38.314466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.314177 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:38.371648 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.369768 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:38.533968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.533935 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm\") pod \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " Nov 03 16:38:38.534221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.534200 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v\" (UniqueName: \"kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v\") pod \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " Nov 03 16:38:38.534398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.534380 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache\") pod \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " Nov 03 16:38:38.534571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.534554 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir\") pod \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " Nov 03 16:38:38.534723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.534708 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs\") pod \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " Nov 03 16:38:38.534920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.534904 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home\") pod \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\" (UID: \"22c115d3-9503-4c0e-a3de-bf23aadce1cd\") " Nov 03 16:38:38.535671 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.535648 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home" pod "22c115d3-9503-4c0e-a3de-bf23aadce1cd" (UID: "22c115d3-9503-4c0e-a3de-bf23aadce1cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:38.535816 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.535798 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache" pod "22c115d3-9503-4c0e-a3de-bf23aadce1cd" (UID: "22c115d3-9503-4c0e-a3de-bf23aadce1cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:38.535918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.535742 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir" pod "22c115d3-9503-4c0e-a3de-bf23aadce1cd" (UID: "22c115d3-9503-4c0e-a3de-bf23aadce1cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:38.546879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.546835 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm" pod "22c115d3-9503-4c0e-a3de-bf23aadce1cd" (UID: "22c115d3-9503-4c0e-a3de-bf23aadce1cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:38.551996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.551858 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs" pod "22c115d3-9503-4c0e-a3de-bf23aadce1cd" (UID: "22c115d3-9503-4c0e-a3de-bf23aadce1cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:38.551996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.551962 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v" pod "22c115d3-9503-4c0e-a3de-bf23aadce1cd" (UID: "22c115d3-9503-4c0e-a3de-bf23aadce1cd"). InnerVolumeSpecName "kube-api-access-hp86v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:38.640466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.640410 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:38:38.640731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.640714 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:38:38.640885 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.640869 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp86v\" (UniqueName: \"kubernetes.io/projected/22c115d3-9503-4c0e-a3de-bf23aadce1cd-kube-api-access-hp86v\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:38:38.641014 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.640999 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:38:38.641199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.641185 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:38:38.641327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:38.641315 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c115d3-9503-4c0e-a3de-bf23aadce1cd-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:38:39.110619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.110569 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:39.110803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.110707 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:39.177556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.177526 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx"] Nov 03 16:38:39.178518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.178495 4539 cpu_manager.go:414] "containerMap: removing container" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerName="main" Nov 03 16:38:39.178518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.178517 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerName="main" Nov 03 16:38:39.178688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.178617 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" containerName="main" Nov 03 16:38:39.182358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.182337 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.205728 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.205707 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"distinct-2-w4-460104-kserve-self-signed-certs\"" Nov 03 16:38:39.210152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.210129 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx"] Nov 03 16:38:39.301230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.301198 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj" Nov 03 16:38:39.316743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.316709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:39.348686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.348654 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.348961 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.348928 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.349166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.349138 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.349239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.349199 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gf9\" (UniqueName: \"kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.349342 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.349314 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.349522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.349365 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.351125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.350541 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj"] Nov 03 16:38:39.380554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.380461 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-b-f7f1d684-kserve-6f5c88cfc7-52gcj"] Nov 03 16:38:39.451049 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.450986 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.451049 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.451054 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.451311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.451101 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.451311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.451125 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.451311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.451154 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.451311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.451173 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gf9\" (UniqueName: \"kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.452174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.452148 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.452454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.452405 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.452631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.452613 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.454740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.454708 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.454988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.454956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.474985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.474941 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gf9\" (UniqueName: \"kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9\") pod \"distinct-2-w4-460104-kserve-7557d5d887-5n8vx\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.494707 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.494683 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:39.806766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:39.806736 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx"] Nov 03 16:38:39.808410 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:38:39.808387 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde7edd38_07cf_4455_ba6d_f2c502452d37.slice/crio-f99e4db4daa046ef4ea3f517c4f6c0dbdd2617b0cbb019c539497df1229ddd3d WatchSource:0}: Error finding container f99e4db4daa046ef4ea3f517c4f6c0dbdd2617b0cbb019c539497df1229ddd3d: Status 404 returned error can't find the container with id f99e4db4daa046ef4ea3f517c4f6c0dbdd2617b0cbb019c539497df1229ddd3d Nov 03 16:38:40.093832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:40.093733 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:40.310493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:40.310454 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" event={"ID":"de7edd38-07cf-4455-ba6d-f2c502452d37","Type":"ContainerStarted","Data":"27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e"} Nov 03 16:38:40.310493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:40.310500 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" event={"ID":"de7edd38-07cf-4455-ba6d-f2c502452d37","Type":"ContainerStarted","Data":"f99e4db4daa046ef4ea3f517c4f6c0dbdd2617b0cbb019c539497df1229ddd3d"} Nov 03 16:38:40.352037 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:40.351934 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podStartSLOduration=1.351918984 podStartE2EDuration="1.351918984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:40.349621271 +0000 UTC m=+1464.042892293" watchObservedRunningTime="2026-11-03 16:38:40.351918984 +0000 UTC m=+1464.045190007" Nov 03 16:38:40.471369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:40.471321 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c115d3-9503-4c0e-a3de-bf23aadce1cd" path="/var/lib/kubelet/pods/22c115d3-9503-4c0e-a3de-bf23aadce1cd/volumes" Nov 03 16:38:40.495390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:40.495355 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:41.704075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:41.703938 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:43.293550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:43.293522 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:43.581324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:43.581218 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:44.978666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:44.978636 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:46.801302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:46.801266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:47.108194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.108098 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg"] Nov 03 16:38:47.112946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.112927 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.129461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.129417 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj"] Nov 03 16:38:47.130847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.130825 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-460104-openshift-default-dockercfg-4t76j\"" Nov 03 16:38:47.132918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.132901 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.133022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133002 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133030 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133148 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133211 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133253 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133322 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133346 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133373 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thrz\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.133538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.133394 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.141102 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.141081 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg"] Nov 03 16:38:47.162200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.162166 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj"] Nov 03 16:38:47.234398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234367 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.234398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234401 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234486 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234511 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234546 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234579 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234609 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234640 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.234677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234672 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234704 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thrz\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234737 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234763 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234789 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9rm\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234880 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234943 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234954 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.234989 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.235016 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.235280 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.235359 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.235380 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.235559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.235393 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.235869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.235849 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.237243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.237215 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.237698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.237678 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.262184 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.262153 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.262317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.262300 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thrz\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.337615 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.337571 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.337815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.337660 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.337815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.337684 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.337815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.337757 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.337993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.337895 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.337993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.337970 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338007 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9rm\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338094 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338104 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338184 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338191 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338283 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338455 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.338996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.338900 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.340501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.340477 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.340859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.340838 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.362687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.362623 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.363870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.363848 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9rm\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.430996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.430969 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:47.451977 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.451951 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:47.751156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.751121 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg"] Nov 03 16:38:47.752141 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:38:47.752115 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa033686_2d07_4941_9839_1e2b6d759464.slice/crio-c38f17447350aaee042a7f1a063f6ebaf2a91f80d214b13e4c4ad579babdb025 WatchSource:0}: Error finding container c38f17447350aaee042a7f1a063f6ebaf2a91f80d214b13e4c4ad579babdb025: Status 404 returned error can't find the container with id c38f17447350aaee042a7f1a063f6ebaf2a91f80d214b13e4c4ad579babdb025 Nov 03 16:38:47.754340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.754312 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:47.754440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.754394 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:47.754485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.754454 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:47.778906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.778884 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj"] Nov 03 16:38:47.782497 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:38:47.782463 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea08502f_d143_4369_b374_ef63f987872e.slice/crio-20f1051bcbed2fcc5518c7b6188fda47ec6afd153e0763ef58417384d172ac6a WatchSource:0}: Error finding container 20f1051bcbed2fcc5518c7b6188fda47ec6afd153e0763ef58417384d172ac6a: Status 404 returned error can't find the container with id 20f1051bcbed2fcc5518c7b6188fda47ec6afd153e0763ef58417384d172ac6a Nov 03 16:38:47.785140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.785103 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:47.785218 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.785187 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:47.785256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:47.785237 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:38:48.364718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.364680 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" event={"ID":"fa033686-2d07-4941-9839-1e2b6d759464","Type":"ContainerStarted","Data":"c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636"} Nov 03 16:38:48.365246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.364744 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" event={"ID":"fa033686-2d07-4941-9839-1e2b6d759464","Type":"ContainerStarted","Data":"c38f17447350aaee042a7f1a063f6ebaf2a91f80d214b13e4c4ad579babdb025"} Nov 03 16:38:48.367127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.367096 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" event={"ID":"ea08502f-d143-4369-b374-ef63f987872e","Type":"ContainerStarted","Data":"25b1e721cb957d461ec873a994fec88dc4ab974d30b858e4634a12e218ea80bd"} Nov 03 16:38:48.367269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.367151 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" event={"ID":"ea08502f-d143-4369-b374-ef63f987872e","Type":"ContainerStarted","Data":"20f1051bcbed2fcc5518c7b6188fda47ec6afd153e0763ef58417384d172ac6a"} Nov 03 16:38:48.432461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.431476 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:48.434716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.434651 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podStartSLOduration=1.434631913 podStartE2EDuration="1.434631913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:48.424062344 +0000 UTC m=+1472.117333367" watchObservedRunningTime="2026-11-03 16:38:48.434631913 +0000 UTC m=+1472.127902935" Nov 03 16:38:48.436563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.436538 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:48.437321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.436989 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:48.454815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.452612 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:48.455007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.454858 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:48.455007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.454898 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:48.484448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.483678 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podStartSLOduration=1.4836542750000001 podStartE2EDuration="1.483654275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:48.468388571 +0000 UTC m=+1472.161659593" watchObservedRunningTime="2026-11-03 16:38:48.483654275 +0000 UTC m=+1472.176925298" Nov 03 16:38:48.612706 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:48.612670 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:49.431874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.431834 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:49.432482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.432451 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:49.455830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.455784 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:49.478592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.478541 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:49.495479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.495454 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:49.584456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.584385 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:49.981241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:49.980538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:50.396138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:50.396108 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:38:50.441450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:50.441340 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:50.441450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:50.441409 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:50.454822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:50.454788 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:50.455095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:50.455062 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:50.469472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:50.468954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:51.435447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:51.434120 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:51.435447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:51.434183 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:51.458563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:51.458529 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:51.459148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:51.459116 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:52.433702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:52.433250 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:52.433702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:52.433312 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:52.436620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:52.436502 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:52.515398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:52.515356 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:52.515909 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:52.515415 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:53.380571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.380523 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:38:53.385865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.385823 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:53.435993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.435957 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:53.436330 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.436283 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:53.443581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.443536 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:53.465463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.459387 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:53.465463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.465371 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:53.560632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.560589 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:53.561276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:53.561229 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:54.214449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.210579 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:54.214449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.210651 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:54.328372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.328332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:54.450837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.450806 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:54.454562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.454515 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:54.542388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.542355 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:54.542705 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:54.542680 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:55.432149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:55.432093 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:38:55.432695 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:55.432185 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 16:38:55.453132 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:55.453092 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:38:55.453304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:55.453172 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 16:38:55.871751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:55.871698 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:56.168824 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:38:56.168736 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:38:57.473635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.473598 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:57.474438 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.474277 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:57.510161 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.510105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:38:57.519453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.514539 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:57.519453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.514595 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:57.573669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.573628 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:57.574379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.574355 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:57.575200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.575182 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:38:57.747466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.747369 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:57.747876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.747857 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:57.748614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:57.748597 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 16:38:59.243320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:38:59.243111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:00.942305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:00.942267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:02.696614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:02.696578 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:04.395922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:04.395883 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:05.477907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:05.477876 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:06.238585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:06.238555 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:08.251975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.251940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:08.366284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.366251 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm"] Nov 03 16:39:08.370017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.369987 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.396916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.396746 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"distinct-2-w2-7519b7-kserve-self-signed-certs\"" Nov 03 16:39:08.400366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.400337 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm"] Nov 03 16:39:08.476965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.476925 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:08.477374 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.477345 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.477503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.477484 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.477588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.477573 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.477635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.477621 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.477701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.477687 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.477766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.477753 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.580127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.580078 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.580986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.580962 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.581214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.581195 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.581376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.581363 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.581519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.581499 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.581695 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.581665 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.581919 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.581512 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.582210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.582186 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.582492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.582402 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.585277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.584799 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.585589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.585570 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.618448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.618153 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4\") pod \"distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:08.685373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:08.685323 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:09.001631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:09.001603 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm"] Nov 03 16:39:09.003119 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:09.003087 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fe3689f_246d_4bb3_b180_73038521aadd.slice/crio-d28ce3607aa31a826265113b6e73fc2a3a5b115630d639e06e8a52960a1c6861 WatchSource:0}: Error finding container d28ce3607aa31a826265113b6e73fc2a3a5b115630d639e06e8a52960a1c6861: Status 404 returned error can't find the container with id d28ce3607aa31a826265113b6e73fc2a3a5b115630d639e06e8a52960a1c6861 Nov 03 16:39:09.431412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:09.431385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:09.756651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:09.756557 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" event={"ID":"5fe3689f-246d-4bb3-b180-73038521aadd","Type":"ContainerStarted","Data":"ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc"} Nov 03 16:39:09.756651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:09.756601 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" event={"ID":"5fe3689f-246d-4bb3-b180-73038521aadd","Type":"ContainerStarted","Data":"d28ce3607aa31a826265113b6e73fc2a3a5b115630d639e06e8a52960a1c6861"} Nov 03 16:39:09.800924 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:09.800862 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podStartSLOduration=1.800841393 podStartE2EDuration="1.800841393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:09.797678966 +0000 UTC m=+1493.490949988" watchObservedRunningTime="2026-11-03 16:39:09.800841393 +0000 UTC m=+1493.494112415" Nov 03 16:39:09.996618 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:09.996585 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:10.686081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:10.686044 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:11.588323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:11.588284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:13.225283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:13.225213 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:14.975208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:14.975177 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:15.380896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.380862 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn"] Nov 03 16:39:15.384723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.384700 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.413894 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.413864 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn"] Nov 03 16:39:15.425508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.425472 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg"] Nov 03 16:39:15.430171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.430145 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.463498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.463464 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg"] Nov 03 16:39:15.554455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554403 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554470 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.554662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554520 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6xt\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554545 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554587 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554604 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554711 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554739 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.554951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554767 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.554951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554829 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.554951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554859 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrq7\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.555165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.554966 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.555165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.555002 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.555165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.555098 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.555258 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.555172 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.555258 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.555201 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.555320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.555289 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.555320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.555309 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.656915 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.656963 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.656991 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6xt\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657017 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657051 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657074 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657133 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657157 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657179 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657268 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657325 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrq7\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657364 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657384 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657406 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657393 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657512 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657537 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657576 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657603 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657668 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657695 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.657845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.657781 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.658578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.658028 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.658578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.658160 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.658578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.658266 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.658578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.658393 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.658578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.658415 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.658891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.658861 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.659956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.659925 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.660401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.660374 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.660401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.660391 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.660734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.660712 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.679247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.679206 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.679401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.679283 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6xt\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:15.679401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.679388 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.680274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.680255 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrq7\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.701468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.701417 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:15.753009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:15.752975 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:16.032910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.032881 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn"] Nov 03 16:39:16.037062 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:16.037033 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda041f70b_c4aa_43b0_8767_e9834c2aa6fc.slice/crio-aa17d54faf72be07b662e34da464fee2c4e55e375565102f2e8e964e3e701a1a WatchSource:0}: Error finding container aa17d54faf72be07b662e34da464fee2c4e55e375565102f2e8e964e3e701a1a: Status 404 returned error can't find the container with id aa17d54faf72be07b662e34da464fee2c4e55e375565102f2e8e964e3e701a1a Nov 03 16:39:16.039757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.039724 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:16.039863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.039805 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:16.039863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.039849 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:16.081406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.081385 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg"] Nov 03 16:39:16.083567 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:16.083540 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0133ecf_85d2_4d69_a6c2_abfe8b6cf861.slice/crio-6f4dc58aa9fd3ed331c9458c8db8669f0ce03ca77ffd1867b11da04137f8aa38 WatchSource:0}: Error finding container 6f4dc58aa9fd3ed331c9458c8db8669f0ce03ca77ffd1867b11da04137f8aa38: Status 404 returned error can't find the container with id 6f4dc58aa9fd3ed331c9458c8db8669f0ce03ca77ffd1867b11da04137f8aa38 Nov 03 16:39:16.086219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.086187 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:16.086304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.086252 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:16.086304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.086281 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:16.316121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.316022 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t"] Nov 03 16:39:16.320988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.320957 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.365241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.365179 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t"] Nov 03 16:39:16.468863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.468827 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.468949 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.468987 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.469064 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.469119 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.469171 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.469233 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.469280 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqdd\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.469485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.469469 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571055 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571116 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqdd\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571175 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571212 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571265 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571289 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571356 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571413 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.571670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571549 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.572020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.571998 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.572116 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.572099 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.572440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.572377 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.572980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.572956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.573322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.573300 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.574185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.574151 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.574276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.574226 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.593250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.593223 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqdd\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.593369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.593339 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.661853 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.661828 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:16.673847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.673824 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:39:16.674042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.674021 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:39:16.766340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.766292 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:16.815492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.814596 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" event={"ID":"a041f70b-c4aa-43b0-8767-e9834c2aa6fc","Type":"ContainerStarted","Data":"774e0c3ed3b6a63973f28d0226c6d62a940ff5831d7a1482cec715bab2e1c908"} Nov 03 16:39:16.815492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.814667 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" event={"ID":"a041f70b-c4aa-43b0-8767-e9834c2aa6fc","Type":"ContainerStarted","Data":"aa17d54faf72be07b662e34da464fee2c4e55e375565102f2e8e964e3e701a1a"} Nov 03 16:39:16.817697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.817669 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" event={"ID":"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861","Type":"ContainerStarted","Data":"b8550d882900a5a408bb16521dbbbb45e478fcfc6be45a49aa6a9eb1cf11c6b6"} Nov 03 16:39:16.817907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.817724 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" event={"ID":"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861","Type":"ContainerStarted","Data":"6f4dc58aa9fd3ed331c9458c8db8669f0ce03ca77ffd1867b11da04137f8aa38"} Nov 03 16:39:16.876405 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.876296 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podStartSLOduration=1.876282245 podStartE2EDuration="1.876282245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:16.874756621 +0000 UTC m=+1500.568027645" watchObservedRunningTime="2026-11-03 16:39:16.876282245 +0000 UTC m=+1500.569553266" Nov 03 16:39:16.944496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:16.944406 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podStartSLOduration=1.9443820509999998 podStartE2EDuration="1.944382051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:16.920031664 +0000 UTC m=+1500.613302689" watchObservedRunningTime="2026-11-03 16:39:16.944382051 +0000 UTC m=+1500.637653074" Nov 03 16:39:17.058673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.058637 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t"] Nov 03 16:39:17.071060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.071020 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:17.071160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.071122 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:17.071201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.071171 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:17.707803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.707764 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:17.731449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.727568 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:17.731449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.727631 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:17.760166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.760124 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:17.762876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.762846 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:17.763118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.763092 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:17.869007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.868967 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" event={"ID":"10caf43a-bae3-4650-93a2-e406713a73ca","Type":"ContainerStarted","Data":"5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3"} Nov 03 16:39:17.869175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.869051 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" event={"ID":"10caf43a-bae3-4650-93a2-e406713a73ca","Type":"ContainerStarted","Data":"7299e31f58cbe6fecc0802a9270edad63a02df5bc5364f1835a692a4b97a9b16"} Nov 03 16:39:17.933477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:17.933391 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podStartSLOduration=1.933368368 podStartE2EDuration="1.933368368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:17.918258322 +0000 UTC m=+1501.611529345" watchObservedRunningTime="2026-11-03 16:39:17.933368368 +0000 UTC m=+1501.626639392" Nov 03 16:39:18.668452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.667770 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:18.690469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.680041 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.690469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.680104 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:18.690469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.686772 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:18.693449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.691816 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:18.705295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.702810 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.705295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.702879 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:18.755448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.754991 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.755448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.755066 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:18.828915 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:18.823793 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:19.195683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.195651 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:19.665447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.664808 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.665447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.664883 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:19.705996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.705963 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.706645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.706617 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:19.757451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.756820 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.757451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:19.756877 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:20.712414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:20.710625 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.712414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:20.710691 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:20.712414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:20.711158 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.712414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:20.711195 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:20.761462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:20.761337 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.761462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:20.761397 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:21.154712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.154664 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.276663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.275530 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:39:21.398450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.397565 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.442540 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.441681 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:39:21.617681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.616532 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.666889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.666854 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.667182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.667154 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:21.708516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.706894 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.708516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.706954 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:21.756171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.756131 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.756817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.756764 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:21.837455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:21.837404 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:22.147450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.146918 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.161248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.161209 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:22.162056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.162033 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:22.414689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.414573 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.415680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.415655 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:39:22.623907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.623862 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:39:22.629447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.627161 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:39:22.636580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.636506 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.668529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.668017 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:39:22.670526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.670498 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.670733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.670696 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:22.738498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.722787 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.738934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.738881 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:22.765610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.765575 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.913449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:22.797497 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:23.163745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.163697 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.363623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.363566 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.387860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.387811 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.426345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.426234 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.625606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.625552 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:39:23.657598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.638014 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.850451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.846898 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.901563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.901515 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:23.901867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.863504 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.902022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.901999 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:23.902142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.863565 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.902263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:23.902245 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:24.050090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.050058 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:24.388451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.364560 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.395863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.395833 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:39:24.456244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.396100 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.456902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.456884 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:39:24.639689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.639559 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.640174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.640144 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 16:39:24.641360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.641337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:24.643086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.643060 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.664864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.664828 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.665143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.665114 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:24.665339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.442886 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.676539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.676501 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.831446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.829349 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.831446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.829407 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:24.831446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.829516 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.831446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.829539 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:24.857373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.857343 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:39:24.895163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.894555 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:39:24.984698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:24.984670 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:39:25.027332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.027296 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:39:25.080478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.080416 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:39:25.662471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.662408 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.662681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.662517 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 16:39:25.702352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.702316 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.702633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.702604 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 16:39:25.754359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.754317 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.754599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:25.754388 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 16:39:26.169972 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:39:26.169849 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:39:26.687749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.687712 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:26.761809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.761782 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:26.763692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.763675 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:26.765110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.765093 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:39:26.781766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.781740 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:26.783038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.783021 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:26.784544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:26.784524 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:39:27.704284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:27.703567 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:27.704284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:27.703658 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.223935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:28.223903 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:28.224556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:28.224531 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:28.225661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:28.225642 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:39:28.499086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:28.498951 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:29.066839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:29.066810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:30.486466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:30.486437 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:32.164479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:32.164411 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:33.848686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:33.848633 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:35.688549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:35.688503 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:35.901574 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:35.901526 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:35.972055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:35.971964 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:36.236324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.236228 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb"] Nov 03 16:39:36.275793 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.275763 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb"] Nov 03 16:39:36.276173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.276156 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351477 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxzt\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351618 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351694 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351730 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351785 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351861 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351905 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.351940 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.378468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.352019 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.456833 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.456893 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.456925 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.456955 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.456989 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxzt\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.457093 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.457153 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.457189 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.457240 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.457824 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.458104 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.458363 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.459075 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.463410 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.493507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.487054 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.593854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.592877 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:36.597097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.597062 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.598092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.598066 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.614738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.614710 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxzt\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:36.768793 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:36.768698 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:37.100670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.100642 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb"] Nov 03 16:39:37.102006 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:37.101979 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ef7b37_de64_408e_b169_8c68842e9cb2.slice/crio-3482146cc841e879ee2d5a726ccba467e8923f380a91e17b3f8911490c1d2ae4 WatchSource:0}: Error finding container 3482146cc841e879ee2d5a726ccba467e8923f380a91e17b3f8911490c1d2ae4: Status 404 returned error can't find the container with id 3482146cc841e879ee2d5a726ccba467e8923f380a91e17b3f8911490c1d2ae4 Nov 03 16:39:37.104533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.104502 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:37.104636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.104568 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:37.104636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.104599 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:37.627548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.627514 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" event={"ID":"e6ef7b37-de64-408e-b169-8c68842e9cb2","Type":"ContainerStarted","Data":"07ea29968aaab97c24d6bcc1c775012364d3440c6a85133c6c6aeccdc60b16c8"} Nov 03 16:39:37.627741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.627579 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" event={"ID":"e6ef7b37-de64-408e-b169-8c68842e9cb2","Type":"ContainerStarted","Data":"3482146cc841e879ee2d5a726ccba467e8923f380a91e17b3f8911490c1d2ae4"} Nov 03 16:39:37.691615 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.691558 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podStartSLOduration=1.691544372 podStartE2EDuration="1.691544372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:37.680141188 +0000 UTC m=+1521.373412210" watchObservedRunningTime="2026-11-03 16:39:37.691544372 +0000 UTC m=+1521.384815394" Nov 03 16:39:37.696090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.696059 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:37.772607 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:37.772570 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:38.770636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:38.770602 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 16:39:38.770831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:38.770659 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 16:39:39.579703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:39.579669 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:39.769204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:39.769159 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 16:39:39.769412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:39.769230 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 16:39:40.769761 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:40.769726 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 16:39:40.770174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:40.769781 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 16:39:41.422448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:41.408099 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:41.769920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:41.769775 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 16:39:41.769920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:41.769831 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 16:39:42.769387 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:42.769349 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 16:39:42.769618 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:42.769454 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 16:39:43.082796 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:43.082753 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:43.769046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:43.769008 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 16:39:43.769275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:43.769061 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 16:39:44.469949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.469915 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k"] Nov 03 16:39:44.473971 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.473953 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.498477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.498411 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-491905-openshift-default-dockercfg-dc6wb\"" Nov 03 16:39:44.503640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.503605 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k"] Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.552748 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.552931 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.552986 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.553015 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrwq\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.553056 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.553113 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.553154 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.553215 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.554471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.553349 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.654666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654624 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.654887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654756 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.654887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654844 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654886 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654922 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654945 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrwq\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.654984 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.655036 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.655072 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.655222 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.655801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.655762 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.656041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.655996 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.656041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.655861 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.656596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.656563 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.657976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.657948 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.658319 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.658298 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.680176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.680143 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.680440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.680396 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrwq\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.773913 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.773886 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:44.774329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.774309 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:44.775083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.775061 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 16:39:44.788148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.788123 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:44.891483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:44.887375 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:45.117465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.117399 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k"] Nov 03 16:39:45.118300 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:45.118274 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56e708e_f97c_44b7_97bd_c31970eab99c.slice/crio-598806f6110d560f6a7fbc962617d836144731edb2d7de7e90e601e92378873d WatchSource:0}: Error finding container 598806f6110d560f6a7fbc962617d836144731edb2d7de7e90e601e92378873d: Status 404 returned error can't find the container with id 598806f6110d560f6a7fbc962617d836144731edb2d7de7e90e601e92378873d Nov 03 16:39:45.120893 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.120857 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:45.120972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.120938 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:45.121010 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.120979 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:45.732445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.731837 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" event={"ID":"e56e708e-f97c-44b7-97bd-c31970eab99c","Type":"ContainerStarted","Data":"e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f"} Nov 03 16:39:45.732445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.731926 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" event={"ID":"e56e708e-f97c-44b7-97bd-c31970eab99c","Type":"ContainerStarted","Data":"598806f6110d560f6a7fbc962617d836144731edb2d7de7e90e601e92378873d"} Nov 03 16:39:45.776315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.776257 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podStartSLOduration=1.77624106 podStartE2EDuration="1.77624106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:45.77432439 +0000 UTC m=+1529.467595412" watchObservedRunningTime="2026-11-03 16:39:45.77624106 +0000 UTC m=+1529.469512082" Nov 03 16:39:45.788677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:45.788641 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:46.266356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:46.266324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:46.717451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:46.714192 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:46.791544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:46.791503 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:39:46.791987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:46.791568 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 16:39:47.280364 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.280318 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k"] Nov 03 16:39:47.284271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.284247 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.356763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.356726 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k"] Nov 03 16:39:47.358631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.358607 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg"] Nov 03 16:39:47.363511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.363486 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.380790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.380651 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg"] Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.399668 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.436687 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.436753 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.436797 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.436910 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkx9\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.436980 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.437034 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.437078 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.437410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.437165 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544072 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544152 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544246 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544321 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkx9\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544363 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544449 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544556 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544590 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544648 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544688 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56g2s\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544718 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544757 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544868 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544954 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.544896 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.545016 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.545073 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.545111 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.545163 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.545504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.545221 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.546343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.545943 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.547062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.547038 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.547218 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.547198 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.551974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.551946 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.553740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.553717 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.566995 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.566968 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkx9\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.567535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.567514 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.598062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.598034 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:47.646405 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646371 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.646573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646462 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56g2s\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.646573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646497 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.646573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646560 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.646670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646590 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.646670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646629 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.646731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646668 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.647390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646813 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.647390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.646880 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.647390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.647183 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.647390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.647232 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.647866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.647453 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.647928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.647882 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.648165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.648136 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.650284 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.650235 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.651168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.651124 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.670528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.670498 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.671757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.671729 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56g2s\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.689781 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.689079 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:47.789798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.788724 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:39:47.789798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.788786 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 16:39:47.965185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.965160 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k"] Nov 03 16:39:47.967364 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:47.967332 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fcd313_2648_4deb_aa60_364a21a54097.slice/crio-c1bae59a44a169c91f7e62c7d425726a2a78a88ead7f7aff0ccffe87c5597da7 WatchSource:0}: Error finding container c1bae59a44a169c91f7e62c7d425726a2a78a88ead7f7aff0ccffe87c5597da7: Status 404 returned error can't find the container with id c1bae59a44a169c91f7e62c7d425726a2a78a88ead7f7aff0ccffe87c5597da7 Nov 03 16:39:47.970178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.970137 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:47.970279 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.970227 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:47.970327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:47.970298 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:48.067036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.067004 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg"] Nov 03 16:39:48.067175 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:39:48.067077 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fe11d1_873c_468f_980a_2787c5b2cdd6.slice/crio-c955726c7ff25e7baa1228c6facc4f3203998942854ca4d5e6dd3544b338d289 WatchSource:0}: Error finding container c955726c7ff25e7baa1228c6facc4f3203998942854ca4d5e6dd3544b338d289: Status 404 returned error can't find the container with id c955726c7ff25e7baa1228c6facc4f3203998942854ca4d5e6dd3544b338d289 Nov 03 16:39:48.069893 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.069861 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:48.069974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.069927 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:48.070019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.069977 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:39:48.396319 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.396176 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:48.768368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.768269 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" event={"ID":"a2fe11d1-873c-468f-980a-2787c5b2cdd6","Type":"ContainerStarted","Data":"d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac"} Nov 03 16:39:48.768368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.768327 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" event={"ID":"a2fe11d1-873c-468f-980a-2787c5b2cdd6","Type":"ContainerStarted","Data":"c955726c7ff25e7baa1228c6facc4f3203998942854ca4d5e6dd3544b338d289"} Nov 03 16:39:48.770182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.770158 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" event={"ID":"37fcd313-2648-4deb-aa60-364a21a54097","Type":"ContainerStarted","Data":"93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d"} Nov 03 16:39:48.770322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.770199 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" event={"ID":"37fcd313-2648-4deb-aa60-364a21a54097","Type":"ContainerStarted","Data":"c1bae59a44a169c91f7e62c7d425726a2a78a88ead7f7aff0ccffe87c5597da7"} Nov 03 16:39:48.788514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.788481 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:39:48.788680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.788532 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 16:39:48.822449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.819349 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podStartSLOduration=1.819330213 podStartE2EDuration="1.819330213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:48.81889595 +0000 UTC m=+1532.512166973" watchObservedRunningTime="2026-11-03 16:39:48.819330213 +0000 UTC m=+1532.512601239" Nov 03 16:39:48.863701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:48.863618 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podStartSLOduration=1.8635728249999999 podStartE2EDuration="1.863572825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:48.861631771 +0000 UTC m=+1532.554902790" watchObservedRunningTime="2026-11-03 16:39:48.863572825 +0000 UTC m=+1532.556843873" Nov 03 16:39:49.601465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.598374 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:39:49.601465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.600017 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:49.601465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.600068 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:49.692815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.692784 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:39:49.695155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.695129 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:49.695406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.695366 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:49.790177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.790143 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:39:49.790501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:49.790471 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 16:39:50.247487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.231496 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:50.601451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.599815 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:50.601451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.599875 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:50.693451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.692829 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:50.693451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.692889 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:50.792447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.791894 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:39:50.792447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:50.791968 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 16:39:51.604452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:51.598442 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:51.604452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:51.598505 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:51.692701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:51.691863 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:51.692701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:51.691923 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:51.790050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:51.790011 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:39:51.790349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:51.790320 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 16:39:52.097060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:52.097027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:52.598954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:52.598917 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:52.599305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:52.599247 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:52.694474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:52.691163 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:52.694474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:52.691226 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:53.090456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.087094 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:53.093353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.093327 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:53.099184 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.099156 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 16:39:53.599305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.599269 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:53.599586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.599491 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:53.703452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.700502 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:53.703452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:53.700608 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:54.660559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.660513 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:54.662045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.661471 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:54.687449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.669059 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:54.687449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.669108 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:54.709248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.701324 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:54.709248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.701381 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:54.776352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.776316 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:54.922506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.921551 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:54.999715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:54.999652 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.028681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.027534 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:39:55.059771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.058619 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.443565 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.443517 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.609826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.609789 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:55.610110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.610080 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:55.661768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.661726 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.696513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.693877 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:55.696513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.693954 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:55.775450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.772618 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd"] Nov 03 16:39:55.810055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.810023 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd"] Nov 03 16:39:55.810386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.810367 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:55.852453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.850709 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:39:55.852453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.850800 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:39:55.932659 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.932611 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.933639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:55.933616 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:39:56.087024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.086984 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.087640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.087617 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:39:56.089713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.089688 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.090217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.090190 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.191346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.191312 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrxc\" (UniqueName: \"kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.216747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.090712 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:39:56.232508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.224321 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.418450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.393103 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.460592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.457666 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.460592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.457947 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrxc\" (UniqueName: \"kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.460592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.443235 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.460592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.459999 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:56.460592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.446500 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:39:56.477554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.461962 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:39:56.805454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.781036 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.236996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.841635 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrxc\" (UniqueName: \"kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc\") pod \"maas-api-revoke-sub-c48007a6-4n4cd\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:57.237212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.888533 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.237368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.237336 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:57.237509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.119819 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.237638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.120044 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:57.237738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.167013 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:39:57.237851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.965861 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.237950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:56.888623 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.238065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.238044 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:57.627356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.595927 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:39:57.673460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.660509 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:39:57.673460 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:39:57.661288 4539 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.198s" Nov 03 16:39:57.673460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.661447 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:39:57.723811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.723777 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.724111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.724066 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:57.786104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.772984 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:57.786104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.775242 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.816469 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:39:57.812723 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:39:57.950510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.925263 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.951317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:57.951268 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:58.184506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.184384 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:58.184506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.184471 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.592560 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.592514 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.608211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.592830 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:39:58.621520 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.592863 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.638473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.626561 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.638473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.631509 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.638473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.637507 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:58.638473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.637595 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.642449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.640517 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:58.642449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.640566 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:58.662571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.661532 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.723358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.723307 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:58.723627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.723593 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:39:58.733003 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.732978 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:39:58.798145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.756156 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:39:58.868920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.866896 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:58.868920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:58.866974 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.275182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.263228 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.304445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.302549 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:59.304445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.302618 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.518827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.516155 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:39:59.610512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.609822 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.651459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.623592 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:59.651459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.624805 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.651459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.630538 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 16:39:59.651459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.630583 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 16:39:59.651459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.630641 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:39:59.721452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.677596 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.722090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.710269 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.724194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.724171 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:59.765369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:39:59.765290 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 16:40:00.119797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.093752 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:40:00.194716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.194683 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd"] Nov 03 16:40:00.366492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.342884 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:40:00.459686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.459532 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:40:00.461126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.460379 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:40:00.461126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.461023 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" event={"ID":"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e","Type":"ContainerStarted","Data":"bf6eaf9222ab99dccc66a68a89ddfd9546da88c1ad452c90cf9633a4da022d78"} Nov 03 16:40:00.498668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.498632 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:40:00.539446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:00.537844 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:01.469809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:01.469778 4539 generic.go:356] "Generic (PLEG): container finished" podID="c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" containerID="c13fbe0d586b130bd9f48284c804577bd085106e58ee65abdb13c3e710a47556" exitCode=0 Nov 03 16:40:01.470324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:01.469868 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" event={"ID":"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e","Type":"ContainerDied","Data":"c13fbe0d586b130bd9f48284c804577bd085106e58ee65abdb13c3e710a47556"} Nov 03 16:40:01.639719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:01.639648 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:01.639951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:01.639726 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:01.692547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:01.692446 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:01.692547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:01.692516 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:02.254178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.254141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:02.448446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.448065 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:40:02.449203 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.449182 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:40:02.450243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.450226 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:40:02.677687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.677660 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:40:02.678807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.678789 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:40:02.679601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.679581 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:40:02.688009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.687985 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:40:02.793591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.793562 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc\" (UniqueName: \"kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc\") pod \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " Nov 03 16:40:02.793808 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.793789 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca\") pod \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " Nov 03 16:40:02.793896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.793836 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca\") pod \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\" (UID: \"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e\") " Nov 03 16:40:02.794309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.794283 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca" pod "c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" (UID: "c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:02.796416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.796378 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc" pod "c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" (UID: "c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e"). InnerVolumeSpecName "kube-api-access-gvrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:02.796735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.796712 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca" pod "c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" (UID: "c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:02.895965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.895888 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvrxc\" (UniqueName: \"kubernetes.io/projected/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-kube-api-access-gvrxc\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:40:02.895965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.895917 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:40:02.895965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:02.895927 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:40:03.483624 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:03.483593 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" Nov 03 16:40:03.483624 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:03.483621 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd" event={"ID":"c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e","Type":"ContainerDied","Data":"bf6eaf9222ab99dccc66a68a89ddfd9546da88c1ad452c90cf9633a4da022d78"} Nov 03 16:40:03.483886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:03.483666 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf6eaf9222ab99dccc66a68a89ddfd9546da88c1ad452c90cf9633a4da022d78" Nov 03 16:40:03.953672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:03.953638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:05.685451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:05.684872 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:07.440450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:07.439453 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.604561 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:07.604510 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.664452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:07.661524 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.247467 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:08.242930 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:40:08.272116 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:08.272069 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.609502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:08.608792 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.655455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:08.620612 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:40:08.684905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:08.680119 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.715832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:08.691923 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:40:09.332490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.249282 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:09.332490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.249389 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.332490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.290544 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.332490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.292226 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:40:09.332490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.293688 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:09.332490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.293748 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.404484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.404415 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:40:09.406580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.406559 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:40:09.483849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.483816 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:09.605467 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.605351 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.692630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.692584 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.727542 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.727504 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:40:09.729934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.729902 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.925537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:09.925407 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:10.492479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.463171 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:40:10.549934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.533081 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.549934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.534130 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.690513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.637652 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.717881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.712329 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:40:10.732940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.730669 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.908698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.908668 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:40:10.909303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:10.909283 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:40:11.070449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:11.070399 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:40:11.070727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:11.070709 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:40:11.070883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:11.070870 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:40:11.071010 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:11.070998 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:40:11.395752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:11.395724 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:13.193000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:13.192934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:15.013810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:15.013773 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:15.781277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:15.781246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:16.071407 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:16.071325 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:17.371515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:17.371482 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:18.832981 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:18.832952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:20.148111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:20.148070 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:21.416312 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.416273 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6"] Nov 03 16:40:21.425454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.419402 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" containerName="revoke-keys" Nov 03 16:40:21.429134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.429110 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" containerName="revoke-keys" Nov 03 16:40:21.429687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.429654 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" containerName="revoke-keys" Nov 03 16:40:21.436962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.436941 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576359 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576416 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2kt\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576536 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576568 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576616 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576640 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576700 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576728 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.576941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.576760 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.580496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.580469 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6"] Nov 03 16:40:21.814479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.809048 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.815862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.815287 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.816351 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.816333 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.816684 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.816667 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.817563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.817543 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.818209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.818190 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.818815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.818796 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.822696 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.818656 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.822863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.818706 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.822996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.818019 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.823368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.823349 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.824104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.823687 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.824235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.823772 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.824384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.824366 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2kt\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.881406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.881185 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:21.928454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.924118 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:22.002679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:21.987251 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2kt\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:22.020800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:22.020763 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:22.108291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:22.107967 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:22.698586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:22.698534 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.732039 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:22.731984 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:40:23.192354 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.176665 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:40:23.192354 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.177339 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.392450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.389203 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:23.448466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.447930 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.478672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.478632 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 16:40:23.593126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.593088 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:23.593438 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.593387 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.853522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.828628 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:40:23.854454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.854221 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:40:23.878468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.872150 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.944576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.878850 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:40:23.944576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.928654 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:23.944576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.928822 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:23.944576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:23.928917 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:24.201384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.190870 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:24.201384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.190968 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.201384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.192261 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.234793 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.234757 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:40:24.422827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.275784 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.422827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.276991 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:40:24.476524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.454162 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:40:24.477142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.464751 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6"] Nov 03 16:40:24.477464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.477417 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:40:24.565634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.565586 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.575573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.575528 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:24.585566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.585517 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.677124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.677084 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.677460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.677406 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 16:40:24.875775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.875692 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.896468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.896411 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.929025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:24.928997 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:40:25.107094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.106852 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" event={"ID":"be7a5237-fbbc-4570-85cf-a47420d9f0ec","Type":"ContainerStarted","Data":"42968c315e722b0c797a76564613d46b1ea3a532f1e5f86bcd8bbef98c54d0bc"} Nov 03 16:40:25.210026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.209971 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:25.300113 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.285723 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:25.417157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.416845 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:40:25.432448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.419152 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:40:25.478922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.476837 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:40:25.478922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.476966 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:25.497491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.479415 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:40:25.498791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.498189 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:40:25.498791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.498472 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:25.572066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:25.571986 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:40:26.000695 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.000664 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm"] Nov 03 16:40:26.004911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.004885 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.017911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.017887 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:40:26.018090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.017960 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:40:26.028017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.027988 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm"] Nov 03 16:40:26.141413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.141382 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.141680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.141564 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87spd\" (UniqueName: \"kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.141680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.141662 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.143101 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.143070 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" event={"ID":"be7a5237-fbbc-4570-85cf-a47420d9f0ec","Type":"ContainerStarted","Data":"21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3"} Nov 03 16:40:26.163311 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:40:26.163278 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:40:26.188802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.188753 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podStartSLOduration=5.1887380610000005 podStartE2EDuration="5.188738061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:26.184711251 +0000 UTC m=+1569.877982272" watchObservedRunningTime="2026-11-03 16:40:26.188738061 +0000 UTC m=+1569.882009083" Nov 03 16:40:26.242593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.242559 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87spd\" (UniqueName: \"kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.242801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.242690 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.243760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.243315 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.243927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.243905 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.245354 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.245332 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.267452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.267367 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87spd\" (UniqueName: \"kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd\") pod \"maas-api-revoke-sub-fb11b61a-qqcwm\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.340012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.318633 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:26.456061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.456028 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:26.642299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.641508 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm"] Nov 03 16:40:26.645392 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:40:26.645359 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1e60e1_92bb_4dd0_a43b_a3f368c2faec.slice/crio-f3b6d31eb2558f22717de95a5b275692588a0f2710e18ef0233d0d4d5c61ce01 WatchSource:0}: Error finding container f3b6d31eb2558f22717de95a5b275692588a0f2710e18ef0233d0d4d5c61ce01: Status 404 returned error can't find the container with id f3b6d31eb2558f22717de95a5b275692588a0f2710e18ef0233d0d4d5c61ce01 Nov 03 16:40:26.766322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:26.766293 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:27.111448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.109743 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:27.120842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.120529 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:40:27.120842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.120590 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 16:40:27.153454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.153386 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" event={"ID":"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec","Type":"ContainerStarted","Data":"5984e7ecd29ef0ddcf92ed159a14a79196494343e7e85d61f177cd00869bda6e"} Nov 03 16:40:27.153662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.153485 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" event={"ID":"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec","Type":"ContainerStarted","Data":"f3b6d31eb2558f22717de95a5b275692588a0f2710e18ef0233d0d4d5c61ce01"} Nov 03 16:40:27.193119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.193067 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" podStartSLOduration=2.193052545 podStartE2EDuration="2.193052545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:27.19068741 +0000 UTC m=+1570.883958425" watchObservedRunningTime="2026-11-03 16:40:27.193052545 +0000 UTC m=+1570.886323568" Nov 03 16:40:27.340308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:27.331453 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:28.115448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:28.114820 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:40:28.115448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:28.114887 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 16:40:28.162159 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:28.162125 4539 generic.go:356] "Generic (PLEG): container finished" podID="2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" containerID="5984e7ecd29ef0ddcf92ed159a14a79196494343e7e85d61f177cd00869bda6e" exitCode=0 Nov 03 16:40:28.162338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:28.162208 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" event={"ID":"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec","Type":"ContainerDied","Data":"5984e7ecd29ef0ddcf92ed159a14a79196494343e7e85d61f177cd00869bda6e"} Nov 03 16:40:29.047008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.046975 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:29.114854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.114811 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:40:29.115172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.115138 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 16:40:29.362279 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.362252 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:29.487253 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.487219 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca\") pod \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " Nov 03 16:40:29.487437 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.487311 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd\" (UniqueName: \"kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd\") pod \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " Nov 03 16:40:29.487549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.487532 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca\") pod \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\" (UID: \"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec\") " Nov 03 16:40:29.487692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.487664 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca" pod "2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" (UID: "2f1e60e1-92bb-4dd0-a43b-a3f368c2faec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:29.488096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.488049 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.490276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.490252 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca" pod "2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" (UID: "2f1e60e1-92bb-4dd0-a43b-a3f368c2faec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:29.490377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.490290 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd" pod "2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" (UID: "2f1e60e1-92bb-4dd0-a43b-a3f368c2faec"). InnerVolumeSpecName "kube-api-access-87spd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:29.589511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.589437 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87spd\" (UniqueName: \"kubernetes.io/projected/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-kube-api-access-87spd\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.589511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.589466 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.635281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.635244 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4"] Nov 03 16:40:29.640620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.640130 4539 cpu_manager.go:414] "containerMap: removing container" podUID="2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" containerName="revoke-keys" Nov 03 16:40:29.640620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.640156 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" containerName="revoke-keys" Nov 03 16:40:29.640620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.640324 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" containerName="revoke-keys" Nov 03 16:40:29.644355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.644329 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.648192 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.648164 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4"] Nov 03 16:40:29.792403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792361 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792450 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792485 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r25x\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792617 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792652 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792816 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792801 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792859 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792898 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.792944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.792924 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.894673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894584 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.894673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894650 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.894898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894684 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.894898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894710 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.894898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894875 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894948 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.894988 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r25x\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895086 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895104 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895125 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895208 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895256 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895466 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.895809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.895790 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.897164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.897144 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.897687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.897664 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.938572 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.938534 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r25x\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.939725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.939697 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:29.962595 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:29.962573 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:30.125452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.115679 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:40:30.125452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.115737 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 16:40:30.176087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.176014 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" Nov 03 16:40:30.176087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.176036 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm" event={"ID":"2f1e60e1-92bb-4dd0-a43b-a3f368c2faec","Type":"ContainerDied","Data":"f3b6d31eb2558f22717de95a5b275692588a0f2710e18ef0233d0d4d5c61ce01"} Nov 03 16:40:30.176087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.176085 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b6d31eb2558f22717de95a5b275692588a0f2710e18ef0233d0d4d5c61ce01" Nov 03 16:40:30.335119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.335084 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4"] Nov 03 16:40:30.337691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.337167 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:30.337691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.337253 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:30.337691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.337303 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:30.857311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:30.857267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:31.113500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.112862 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:40:31.113500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.112951 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 16:40:31.185171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.185143 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" event={"ID":"d10e1c18-810f-4ee0-affb-934923f0a9d9","Type":"ContainerStarted","Data":"1875d1f042151c96090c809e2ed9bb07d38695de8fc6fda5ea8e3c933e028026"} Nov 03 16:40:31.185419 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.185404 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" event={"ID":"d10e1c18-810f-4ee0-affb-934923f0a9d9","Type":"ContainerStarted","Data":"b2342e5fd1130c7de0c8b347397e4d134efefe9751c868627a1221a6012a9a77"} Nov 03 16:40:31.962995 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.962953 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:31.964767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.964738 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:40:31.964899 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:31.964788 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 16:40:32.116450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:32.110786 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:40:32.116450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:32.110844 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 16:40:32.758215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:32.758186 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:32.963252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:32.963208 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:40:32.963813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:32.963273 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 16:40:33.242792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:33.242763 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:33.244368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:33.244337 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:33.245617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:33.245599 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 16:40:33.299480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:33.299403 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podStartSLOduration=4.299388037 podStartE2EDuration="4.299388037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:31.266176906 +0000 UTC m=+1574.959447929" watchObservedRunningTime="2026-11-03 16:40:33.299388037 +0000 UTC m=+1576.992659059" Nov 03 16:40:33.963105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:33.963069 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:40:33.963292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:33.963134 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 16:40:34.810376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:34.810341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:34.963171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:34.963138 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:40:34.963355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:34.963235 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 16:40:35.778771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:35.778739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:35.963149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:35.963108 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:40:35.963332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:35.963172 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 16:40:36.869924 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:36.869892 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:36.963706 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:36.963667 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:40:36.963923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:36.963750 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 16:40:37.040834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:37.040798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:37.192200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:37.192103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:37.980900 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:37.980873 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:37.981379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:37.981334 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:37.983143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:37.983119 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:40:38.141002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.140967 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:38.289795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.289760 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:38.738136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.738050 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz"] Nov 03 16:40:38.753142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.753116 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.773503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.769694 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8650a6e8-openshift-default-dockercfg-dq7gn\"" Nov 03 16:40:38.775673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.775648 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz"] Nov 03 16:40:38.845121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.845093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:38.910349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910309 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vrx\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910377 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910514 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910552 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910595 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910655 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910689 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910719 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.910966 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.910849 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:38.997095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:38.996967 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:39.012044 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012011 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012141 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vrx\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012191 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012226 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012259 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012286 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012365 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012405 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012453 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012515 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.012911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.012886 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.013121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.013102 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.013571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.013547 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.015041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.014094 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.015748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.015725 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.016910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.016884 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.052032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.051965 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.061068 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.061034 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vrx\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.086771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.086732 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:39.418863 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:40:39.418829 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e27eaf5_a18a_4bef_b46b_aa4ac77ee326.slice/crio-0418d05636e18447cff6c65919b75309bdca85cf06c65b0108ad0393c0500518 WatchSource:0}: Error finding container 0418d05636e18447cff6c65919b75309bdca85cf06c65b0108ad0393c0500518: Status 404 returned error can't find the container with id 0418d05636e18447cff6c65919b75309bdca85cf06c65b0108ad0393c0500518 Nov 03 16:40:39.419991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.419961 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz"] Nov 03 16:40:39.421991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.421958 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:39.422093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.422037 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:39.422093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:39.422083 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:40.284720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:40.284688 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" event={"ID":"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326","Type":"ContainerStarted","Data":"a8b0be3638527d0192e26b5c76a250e77079b4885ade2f343b6645e8a76fb798"} Nov 03 16:40:40.285137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:40.284739 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" event={"ID":"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326","Type":"ContainerStarted","Data":"0418d05636e18447cff6c65919b75309bdca85cf06c65b0108ad0393c0500518"} Nov 03 16:40:40.351984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:40.351928 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podStartSLOduration=2.351913462 podStartE2EDuration="2.351913462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:40.338194041 +0000 UTC m=+1584.031465066" watchObservedRunningTime="2026-11-03 16:40:40.351913462 +0000 UTC m=+1584.045184481" Nov 03 16:40:40.733668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:40.733574 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:41.087999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:41.087964 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:41.093614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:41.093593 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:41.291824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:41.291793 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:41.292795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:41.292774 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:40:42.554856 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:42.554828 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:42.976509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:42.976390 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:43.159086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:43.159049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:44.286175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:44.286143 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:44.786504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:44.783108 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d"] Nov 03 16:40:44.789539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:44.789514 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:44.818448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:44.815011 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d"] Nov 03 16:40:45.058506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.055275 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:45.147452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.147403 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.147857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.147830 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.148108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.148089 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.148305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.148286 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsr6k\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.148552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.148513 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.148764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.148747 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.148931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.148915 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.149123 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.149105 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.150196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.150175 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.261581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.261547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:45.262804 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.262775 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.279004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.278803 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.279230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.278991 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.279513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.279493 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.279680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.279665 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsr6k\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.279815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.279803 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.279941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.279928 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.280039 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.280025 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.280175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.280163 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.280283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.280271 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.280697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.279255 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.281466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.281446 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.282363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.282343 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.288567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.283231 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.288567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.287545 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.288567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.288362 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.306225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.306196 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.359409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.359326 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsr6k\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.497399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.497369 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:45.593454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:45.593402 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:46.028524 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:40:46.028494 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5586617a_e287_4fdd_9d22_ad436f7f385c.slice/crio-0fa0182f68ded1a02e9ba1a7731c3bb45d89b06b683a88793d60011ecab87521 WatchSource:0}: Error finding container 0fa0182f68ded1a02e9ba1a7731c3bb45d89b06b683a88793d60011ecab87521: Status 404 returned error can't find the container with id 0fa0182f68ded1a02e9ba1a7731c3bb45d89b06b683a88793d60011ecab87521 Nov 03 16:40:46.028938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.028918 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d"] Nov 03 16:40:46.031214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.031182 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:46.031317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.031248 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:46.031317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.031282 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:40:46.484878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.484789 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" event={"ID":"5586617a-e287-4fdd-9d22-ad436f7f385c","Type":"ContainerStarted","Data":"b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264"} Nov 03 16:40:46.484878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.484860 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" event={"ID":"5586617a-e287-4fdd-9d22-ad436f7f385c","Type":"ContainerStarted","Data":"0fa0182f68ded1a02e9ba1a7731c3bb45d89b06b683a88793d60011ecab87521"} Nov 03 16:40:46.498628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.498596 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:46.545491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:46.539302 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podStartSLOduration=2.539283032 podStartE2EDuration="2.539283032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:46.5304896 +0000 UTC m=+1590.223760623" watchObservedRunningTime="2026-11-03 16:40:46.539283032 +0000 UTC m=+1590.232554055" Nov 03 16:40:47.508447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:47.507552 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:40:47.508447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:47.507648 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 16:40:47.743447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:47.741712 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:48.130451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:48.129933 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:48.498544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:48.498462 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:40:48.498544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:48.498520 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 16:40:49.515580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:49.515545 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:40:49.516195 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:49.516157 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 16:40:50.498600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:50.498565 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:40:50.498773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:50.498623 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 16:40:51.072130 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:51.072059 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:51.240923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:51.240889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:51.498581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:51.498490 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:40:51.498581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:51.498547 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 16:40:51.507982 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:51.507900 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:52.499166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:52.499133 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:40:52.499719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:52.499192 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 16:40:53.221091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:53.221030 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:53.424323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:53.424291 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:53.554868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:53.554832 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:53.555567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:53.555343 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:53.556486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:53.556466 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:40:53.636331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:53.636295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:54.632093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:54.632058 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:55.029848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.029819 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk"] Nov 03 16:40:55.034136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.034118 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.059487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.052246 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8650a6e8-serving-cert\"" Nov 03 16:40:55.075336 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.075304 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:55.075686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.075349 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk"] Nov 03 16:40:55.120412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.120367 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.120647 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.120472 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nph52\" (UniqueName: \"kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.120770 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.120750 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.120898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.120881 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.222132 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.222097 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.222355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.222138 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nph52\" (UniqueName: \"kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.222355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.222294 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.222355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.222336 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.223011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.222986 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.225057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.225033 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.225226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.225209 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.243951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.243922 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nph52\" (UniqueName: \"kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.257579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.257536 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:55.372775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.372683 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:55.413356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.413264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:55.899494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.899463 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk"] Nov 03 16:40:55.900950 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:40:55.900922 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cb3eb_7451_4f94_aac1_d3d6f712d468.slice/crio-b54de4d2a942f28203e99050bd33aac344f1063b65dcea43aa7fc5a9a367792d WatchSource:0}: Error finding container b54de4d2a942f28203e99050bd33aac344f1063b65dcea43aa7fc5a9a367792d: Status 404 returned error can't find the container with id b54de4d2a942f28203e99050bd33aac344f1063b65dcea43aa7fc5a9a367792d Nov 03 16:40:55.902771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:55.902751 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:40:56.159951 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:40:56.159859 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:40:56.682815 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:56.682778 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" event={"ID":"2b6cb3eb-7451-4f94-aac1-d3d6f712d468","Type":"ContainerStarted","Data":"eac57e304eacd61f400a2f34cb00955a2affbeb6ca91e557e83f23c548ad2978"} Nov 03 16:40:56.683018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:56.682841 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" event={"ID":"2b6cb3eb-7451-4f94-aac1-d3d6f712d468","Type":"ContainerStarted","Data":"b54de4d2a942f28203e99050bd33aac344f1063b65dcea43aa7fc5a9a367792d"} Nov 03 16:40:56.683018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:56.682890 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:40:56.739791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:56.739673 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" podStartSLOduration=1.253018845 podStartE2EDuration="1.73965429s" totalImagesPullingTime="486.635445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:56.736501622 +0000 UTC m=+1600.429772647" watchObservedRunningTime="2026-11-03 16:40:56.73965429 +0000 UTC m=+1600.432925313" Nov 03 16:40:57.393234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:57.393185 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:57.699604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:57.699499 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:40:58.975277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:40:58.975232 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:00.679175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:00.679141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:02.532450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:02.531998 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:02.714671 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:02.714640 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:41:03.432610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:03.432561 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:03.577575 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:03.577530 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.347599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:04.347552 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.486553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:04.486509 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.660665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:04.636728 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.690474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:04.661332 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:41:04.719851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:04.719804 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 16:41:05.096179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:05.095409 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:41:05.958566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:05.958529 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:07.637181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.637141 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p"] Nov 03 16:41:07.641531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.641508 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.661810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.661788 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9501ab-serving-cert\"" Nov 03 16:41:07.679582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.679554 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p"] Nov 03 16:41:07.700886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.700840 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.701049 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.701017 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.701125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.701095 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.701312 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.701276 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cb8\" (UniqueName: \"kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.802392 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.802630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.802524 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.802630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.802551 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.802630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.802601 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cb8\" (UniqueName: \"kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.802786 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:41:07.802696 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-serving-cert: secret "maas-api-e2e-api-a-9501ab-serving-cert" not found Nov 03 16:41:07.802786 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:41:07.802766 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls podName:27d31a27-c3c5-4971-83af-3d7d49308e9d nodeName:}" failed. No retries permitted until 2026-11-03 16:41:08.302745142 +0000 UTC m=+1611.996016150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls") pod "maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" (UID: "27d31a27-c3c5-4971-83af-3d7d49308e9d") : secret "maas-api-e2e-api-a-9501ab-serving-cert" not found Nov 03 16:41:07.803524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.803497 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.805269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.805245 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.825115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.825088 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cb8\" (UniqueName: \"kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:07.831175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:07.831138 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:08.081065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:08.081035 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:08.310905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:08.310866 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:08.314292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:08.314262 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\") pod \"maas-api-e2e-api-a-9501ab-6d956bc97-4s59p\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:08.556471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:08.556418 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:08.910458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:08.907924 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p"] Nov 03 16:41:09.174347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:09.174260 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" event={"ID":"27d31a27-c3c5-4971-83af-3d7d49308e9d","Type":"ContainerStarted","Data":"294118d1c0f2cf58198ce1c17dc43b1d874efb7b3618d87b5107d6cb7e1e49b6"} Nov 03 16:41:09.783723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:09.783688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:10.073500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:10.073401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:10.183376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:10.183340 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" event={"ID":"27d31a27-c3c5-4971-83af-3d7d49308e9d","Type":"ContainerStarted","Data":"d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0"} Nov 03 16:41:10.183655 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:10.183636 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:10.228111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:10.228051 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" podStartSLOduration=2.839594914 podStartE2EDuration="3.228034983s" totalImagesPullingTime="388.440069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:10.226206043 +0000 UTC m=+1613.919477064" watchObservedRunningTime="2026-11-03 16:41:10.228034983 +0000 UTC m=+1613.921306004" Nov 03 16:41:11.590412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:11.590363 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:12.788430 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:12.788379 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:13.494729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:13.494693 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:15.312085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:15.312055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:16.196530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:16.196502 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:41:16.735754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:16.735728 4539 scope.go:122] "RemoveContainer" containerID="a52c958d48644213229ce68960b16994d514778690eff556876e70f278da2f1c" Nov 03 16:41:17.270442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:17.270401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:19.422727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:19.422675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:20.756386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:20.756354 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:21.437223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:21.437112 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:23.292184 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:23.292147 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:24.005480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.005443 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr"] Nov 03 16:41:24.012715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.012686 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.048661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.048627 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr"] Nov 03 16:41:24.088503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.088465 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.088712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.088550 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.088712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.088656 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.088833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.088709 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.088893 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.088828 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.088958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.088941 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.089038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.089020 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.089094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.089060 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.089155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.089094 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrhm\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190444 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190513 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190568 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190630 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190699 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190787 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.190857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190831 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrhm\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190980 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.190993 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191183 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.191067 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.191265 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.191380 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.191445 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.191599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.191527 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.192982 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.192959 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.193875 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.193852 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.214363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.214334 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.214848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.214821 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrhm\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.341812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.341717 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:24.688561 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:41:24.688515 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f4ba74_0cc1_4ab0_a165_c522537165da.slice/crio-465e20675a5047a47b92033e2ddf0444ba536f35cbb836d7b352d7d197c6249b WatchSource:0}: Error finding container 465e20675a5047a47b92033e2ddf0444ba536f35cbb836d7b352d7d197c6249b: Status 404 returned error can't find the container with id 465e20675a5047a47b92033e2ddf0444ba536f35cbb836d7b352d7d197c6249b Nov 03 16:41:24.689748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.689721 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr"] Nov 03 16:41:24.691498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.691469 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:41:24.691576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.691531 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:41:24.691576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:24.691558 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:41:25.115178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.115106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:25.307510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.307473 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" event={"ID":"42f4ba74-0cc1-4ab0-a165-c522537165da","Type":"ContainerStarted","Data":"63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027"} Nov 03 16:41:25.307707 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.307554 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" event={"ID":"42f4ba74-0cc1-4ab0-a165-c522537165da","Type":"ContainerStarted","Data":"465e20675a5047a47b92033e2ddf0444ba536f35cbb836d7b352d7d197c6249b"} Nov 03 16:41:25.342643 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.342612 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:25.344146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.344110 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:25.344278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.344168 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:25.363347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:25.363290 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podStartSLOduration=2.363274096 podStartE2EDuration="2.363274096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:25.361554027 +0000 UTC m=+1629.054825049" watchObservedRunningTime="2026-11-03 16:41:25.363274096 +0000 UTC m=+1629.056545118" Nov 03 16:41:26.163794 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:41:26.163758 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:41:26.343399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:26.343360 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:26.343800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:26.343449 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:27.124451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:27.124266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:27.332630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:27.332591 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:41:27.332778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:27.332654 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:27.342687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:27.342660 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:27.342847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:27.342705 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:28.342713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.342682 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:28.343190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.342738 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:28.349851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.349825 4539 generic.go:356] "Generic (PLEG): container finished" podID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerID="a8b0be3638527d0192e26b5c76a250e77079b4885ade2f343b6645e8a76fb798" exitCode=0 Nov 03 16:41:28.349980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.349901 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" event={"ID":"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326","Type":"ContainerDied","Data":"a8b0be3638527d0192e26b5c76a250e77079b4885ade2f343b6645e8a76fb798"} Nov 03 16:41:28.350355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.350339 4539 scope.go:122] "RemoveContainer" containerID="a8b0be3638527d0192e26b5c76a250e77079b4885ade2f343b6645e8a76fb798" Nov 03 16:41:28.351914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.351891 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:41:28.351985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.351957 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:41:28.352022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:28.351989 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:41:29.029489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:29.029409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:29.350506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:29.349938 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:29.350506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:29.350016 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:29.371312 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:29.371275 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" event={"ID":"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326","Type":"ContainerStarted","Data":"75e8f42b37db4b9c8583fa73ee324d29b1e2e67590d34f92e6b2886d1d59a7a5"} Nov 03 16:41:29.475682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:29.475646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:30.078448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.036897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:30.103491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.087188 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:41:30.113454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.087248 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:41:30.121449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.117534 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 16:41:30.359318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.359217 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:30.381551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.374262 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:30.452449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:30.452013 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:31.087990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:31.087954 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:41:31.088202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:31.088014 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 16:41:31.190271 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:31.190238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:31.371508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:31.364632 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:41:31.371508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:31.364703 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 16:41:32.087366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:32.087330 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:41:32.087612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:32.087383 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 16:41:32.347133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:32.347043 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:32.347387 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:32.347367 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:32.348280 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:32.348262 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:41:33.087373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:33.087336 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:41:33.087803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:33.087391 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 16:41:33.331133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:33.331095 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:34.087221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:34.087183 4539 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 16:41:34.087466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:34.087238 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 16:41:34.418353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:34.417331 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:34.650154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:34.650115 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:35.091521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:35.091489 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:41:35.092025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:35.092011 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:41:35.092833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:35.092812 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:41:35.305066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:35.305037 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:35.652009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:35.651977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:36.287786 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:36.287730 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:36.530749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:36.530311 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:37.320750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:37.320680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:39.211379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:39.211332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:40.911624 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:40.911583 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:42.693367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:42.693304 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:44.422089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:44.420512 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:46.531449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:46.531402 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:48.245178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.245140 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx"] Nov 03 16:41:48.250234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.250212 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.277493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.277414 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d89163-serving-cert\"" Nov 03 16:41:48.286389 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.286362 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx"] Nov 03 16:41:48.369386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.369352 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.369631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.369565 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67qj\" (UniqueName: \"kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.369631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.369626 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.369747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.369710 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.471009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.470958 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67qj\" (UniqueName: \"kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.471236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.471074 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.471236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.471126 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.471236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.471198 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.472193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.472168 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.474547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.474522 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.475870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.475841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:48.479065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.479042 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.492971 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.492945 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67qj\" (UniqueName: \"kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj\") pod \"maas-api-e2e-api-b-d89163-698c747997-lfjvx\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.567242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.567213 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:48.621288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.621215 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:48.876566 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:41:48.876515 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4449a1_880c_47b4_bfa5_9a3caeb5f613.slice/crio-4ecbc634ecbbd6f711393d704a0d4804027f40a2cb4af120f3ac9c5563e90b27 WatchSource:0}: Error finding container 4ecbc634ecbbd6f711393d704a0d4804027f40a2cb4af120f3ac9c5563e90b27: Status 404 returned error can't find the container with id 4ecbc634ecbbd6f711393d704a0d4804027f40a2cb4af120f3ac9c5563e90b27 Nov 03 16:41:48.876726 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:48.876580 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx"] Nov 03 16:41:49.576953 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:49.576917 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" event={"ID":"4d4449a1-880c-47b4-bfa5-9a3caeb5f613","Type":"ContainerStarted","Data":"7a4315b07adfc8f2cb794bfc6c3bc119aa0bd5105277fd32e44da9ef784523f1"} Nov 03 16:41:49.577326 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:49.576973 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" event={"ID":"4d4449a1-880c-47b4-bfa5-9a3caeb5f613","Type":"ContainerStarted","Data":"4ecbc634ecbbd6f711393d704a0d4804027f40a2cb4af120f3ac9c5563e90b27"} Nov 03 16:41:49.577326 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:49.577109 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:49.617486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:49.617408 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" podStartSLOduration=1.288645773 podStartE2EDuration="1.617392677s" totalImagesPullingTime="328.746904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:49.616659492 +0000 UTC m=+1653.309930515" watchObservedRunningTime="2026-11-03 16:41:49.617392677 +0000 UTC m=+1653.310663702" Nov 03 16:41:50.279870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:50.279269 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:51.791998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:51.791957 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:53.377314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:53.377282 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:55.068251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:55.068211 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:55.588905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:55.588877 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:41:56.159249 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:41:56.159208 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:41:56.637820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:56.637764 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:58.321983 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:58.321949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:41:59.973752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:41:59.973700 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:01.566205 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:01.566175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:03.152043 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:03.152002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:05.061370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:05.061332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:06.672158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:06.672120 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:08.370314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:08.370274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:10.212041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:10.211992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:10.394569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:10.394531 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:10.578452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:10.572233 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:10.930325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:10.930238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:12.261380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:12.261326 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:13.938830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:13.938788 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:15.609289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:15.609249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:17.262320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:17.262275 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:17.812791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:17.812746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:17.975578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:17.975548 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:18.971927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:18.971899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:19.152571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.152540 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:19.727502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.727366 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:19.733525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.733497 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss"] Nov 03 16:42:19.741719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.741692 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss"] Nov 03 16:42:19.742041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.741986 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.760492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.758895 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6fa98e-openshift-default-dockercfg-vl6kx\"" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939608 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939653 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdzk\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939726 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939759 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939788 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939817 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939865 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.939972 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.940206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.940124 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:19.966186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:19.966145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:20.042967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.042903 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.042992 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043047 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdzk\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043087 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043129 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043171 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043216 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043272 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043317 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.043893 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.043747 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.044360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.044334 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.044484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.044450 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.044678 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.044655 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.045310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.045263 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.050150 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.050112 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.056190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.056155 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.067960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.067926 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdzk\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.071586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.071558 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token\") pod \"e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.143163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.143123 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:20.484382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.484357 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss"] Nov 03 16:42:20.488708 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:42:20.487167 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099be21c_610d_4c5e_82ee_b3689ba56ef8.slice/crio-f8ac81a3701770d8c1d47218f38aaeeb97bffeef5d25a993d8463ecc835ae062 WatchSource:0}: Error finding container f8ac81a3701770d8c1d47218f38aaeeb97bffeef5d25a993d8463ecc835ae062: Status 404 returned error can't find the container with id f8ac81a3701770d8c1d47218f38aaeeb97bffeef5d25a993d8463ecc835ae062 Nov 03 16:42:20.490399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.490370 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:42:20.490508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.490460 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:42:20.490508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.490490 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:42:20.930944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.930907 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" event={"ID":"099be21c-610d-4c5e-82ee-b3689ba56ef8","Type":"ContainerStarted","Data":"45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea"} Nov 03 16:42:20.931161 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.931028 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" event={"ID":"099be21c-610d-4c5e-82ee-b3689ba56ef8","Type":"ContainerStarted","Data":"f8ac81a3701770d8c1d47218f38aaeeb97bffeef5d25a993d8463ecc835ae062"} Nov 03 16:42:20.978715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:20.978664 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podStartSLOduration=1.9786500930000002 podStartE2EDuration="1.978650093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:20.975798825 +0000 UTC m=+1684.669069847" watchObservedRunningTime="2026-11-03 16:42:20.978650093 +0000 UTC m=+1684.671921113" Nov 03 16:42:21.144275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:21.144243 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:21.150285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:21.150261 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:21.185560 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:21.185473 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:21.938182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:21.938150 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:21.939131 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:21.939109 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:42:22.875601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:22.875569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:23.519597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:23.519566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:23.685374 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:23.685302 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:24.547485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:24.547418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:24.749631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:24.749568 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:24.912019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:24.911925 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:25.334539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:25.334503 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:25.502332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:25.502300 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:26.157494 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:26.157457 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:42:26.233681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.233650 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p"] Nov 03 16:42:26.233890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.233865 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" podUID="27d31a27-c3c5-4971-83af-3d7d49308e9d" containerName="maas-api" containerID="cri-o://d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0" gracePeriod=30 Nov 03 16:42:26.353502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.353287 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:26.694565 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.694534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:26.705041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.704530 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:42:26.807454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.787818 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs\") pod \"27d31a27-c3c5-4971-83af-3d7d49308e9d\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " Nov 03 16:42:26.807454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.802253 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca\") pod \"27d31a27-c3c5-4971-83af-3d7d49308e9d\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " Nov 03 16:42:26.807454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.802331 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8\" (UniqueName: \"kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8\") pod \"27d31a27-c3c5-4971-83af-3d7d49308e9d\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " Nov 03 16:42:26.807454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.802484 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\") pod \"27d31a27-c3c5-4971-83af-3d7d49308e9d\" (UID: \"27d31a27-c3c5-4971-83af-3d7d49308e9d\") " Nov 03 16:42:26.814456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.802078 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs" pod "27d31a27-c3c5-4971-83af-3d7d49308e9d" (UID: "27d31a27-c3c5-4971-83af-3d7d49308e9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:42:26.814456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.811793 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca" pod "27d31a27-c3c5-4971-83af-3d7d49308e9d" (UID: "27d31a27-c3c5-4971-83af-3d7d49308e9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:26.911526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.906796 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8" pod "27d31a27-c3c5-4971-83af-3d7d49308e9d" (UID: "27d31a27-c3c5-4971-83af-3d7d49308e9d"). InnerVolumeSpecName "kube-api-access-j2cb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:26.911526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.911044 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cb8\" (UniqueName: \"kubernetes.io/projected/27d31a27-c3c5-4971-83af-3d7d49308e9d-kube-api-access-j2cb8\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:26.911526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.911075 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:26.911526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.911091 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d31a27-c3c5-4971-83af-3d7d49308e9d-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:26.912939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.912910 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls" pod "27d31a27-c3c5-4971-83af-3d7d49308e9d" (UID: "27d31a27-c3c5-4971-83af-3d7d49308e9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:42:26.982297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.982207 4539 generic.go:356] "Generic (PLEG): container finished" podID="27d31a27-c3c5-4971-83af-3d7d49308e9d" containerID="d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0" exitCode=0 Nov 03 16:42:26.982297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.982258 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" Nov 03 16:42:26.982524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.982295 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" event={"ID":"27d31a27-c3c5-4971-83af-3d7d49308e9d","Type":"ContainerDied","Data":"d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0"} Nov 03 16:42:26.982524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.982364 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p" event={"ID":"27d31a27-c3c5-4971-83af-3d7d49308e9d","Type":"ContainerDied","Data":"294118d1c0f2cf58198ce1c17dc43b1d874efb7b3618d87b5107d6cb7e1e49b6"} Nov 03 16:42:26.982524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.982393 4539 scope.go:122] "RemoveContainer" containerID="d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0" Nov 03 16:42:26.995270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.995243 4539 scope.go:122] "RemoveContainer" containerID="d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0" Nov 03 16:42:26.995698 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:26.995678 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0\": container with ID starting with d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0 not found: ID does not exist" containerID="d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0" Nov 03 16:42:26.995789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:26.995709 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0"} err="failed to get container status \"d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0\": rpc error: code = NotFound desc = could not find container \"d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0\": container with ID starting with d2075f4bcef86683e91ac72f17601ca37d57841f99e224832d8fe41bc975e1b0 not found: ID does not exist" Nov 03 16:42:27.012573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:27.012544 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d31a27-c3c5-4971-83af-3d7d49308e9d-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:27.030579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:27.030546 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p"] Nov 03 16:42:27.040520 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:27.040495 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9501ab-6d956bc97-4s59p"] Nov 03 16:42:27.303929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:27.303899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:28.479462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:28.479411 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d31a27-c3c5-4971-83af-3d7d49308e9d" path="/var/lib/kubelet/pods/27d31a27-c3c5-4971-83af-3d7d49308e9d/volumes" Nov 03 16:42:28.862269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:28.862236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:30.331493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:30.331460 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:30.533064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:30.533032 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:31.274365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:31.274328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:31.555286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:31.555205 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:31.709489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:31.709455 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:32.407785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:32.407751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:32.567058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:32.567014 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:34.164362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:34.164326 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:35.789416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:35.789378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:37.530717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.530670 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2"] Nov 03 16:42:37.531409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.531391 4539 cpu_manager.go:414] "containerMap: removing container" podUID="27d31a27-c3c5-4971-83af-3d7d49308e9d" containerName="maas-api" Nov 03 16:42:37.531509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.531411 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d31a27-c3c5-4971-83af-3d7d49308e9d" containerName="maas-api" Nov 03 16:42:37.531550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.531518 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d31a27-c3c5-4971-83af-3d7d49308e9d" containerName="maas-api" Nov 03 16:42:37.536628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.536608 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.568717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.568679 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6fa98e-serving-cert\"" Nov 03 16:42:37.575760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.575732 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2"] Nov 03 16:42:37.672321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.672282 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.672563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.672543 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.672623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.672574 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx95d\" (UniqueName: \"kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.672764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.672744 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.758651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.758617 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.775958 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.776050 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.776275 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.776301 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx95d\" (UniqueName: \"kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:37.776834 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-serving-cert: secret "maas-api-e2e-api-a-6fa98e-serving-cert" not found Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:37.776907 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls podName:9fddbed9-6b75-4677-a72f-2a1864e6ec7b nodeName:}" failed. No retries permitted until 2026-11-03 16:42:38.27688734 +0000 UTC m=+1701.970158343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls") pod "maas-api-e2e-api-a-6fa98e-74499d6898-85px2" (UID: "9fddbed9-6b75-4677-a72f-2a1864e6ec7b") : secret "maas-api-e2e-api-a-6fa98e-serving-cert" not found Nov 03 16:42:37.779800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.778010 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.803483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.803405 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx95d\" (UniqueName: \"kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.804146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.804119 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:37.897980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:37.897947 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:38.051941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.051911 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:38.284345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.284306 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:38.287122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.287094 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\") pod \"maas-api-e2e-api-a-6fa98e-74499d6898-85px2\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:38.453595 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.453565 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:38.540240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.536231 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:38.726744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.726715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:38.839213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:38.839174 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2"] Nov 03 16:42:38.839607 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:42:38.839586 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fddbed9_6b75_4677_a72f_2a1864e6ec7b.slice/crio-cbcd9b87a1d1325fa8f4b41ab55643983209d52c555f7094221b18ea19066968 WatchSource:0}: Error finding container cbcd9b87a1d1325fa8f4b41ab55643983209d52c555f7094221b18ea19066968: Status 404 returned error can't find the container with id cbcd9b87a1d1325fa8f4b41ab55643983209d52c555f7094221b18ea19066968 Nov 03 16:42:39.137507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:39.130113 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" event={"ID":"9fddbed9-6b75-4677-a72f-2a1864e6ec7b","Type":"ContainerStarted","Data":"cbcd9b87a1d1325fa8f4b41ab55643983209d52c555f7094221b18ea19066968"} Nov 03 16:42:39.834676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:39.834642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:40.031391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:40.031352 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:40.140008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:40.139918 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" event={"ID":"9fddbed9-6b75-4677-a72f-2a1864e6ec7b","Type":"ContainerStarted","Data":"ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d"} Nov 03 16:42:40.140186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:40.140068 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:40.181371 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:40.181029 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" podStartSLOduration=2.760353718 podStartE2EDuration="3.181010078s" totalImagesPullingTime="420.65636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:40.179332416 +0000 UTC m=+1703.872603439" watchObservedRunningTime="2026-11-03 16:42:40.181010078 +0000 UTC m=+1703.874281103" Nov 03 16:42:41.229547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:41.229509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:41.904881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:41.904848 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:43.580382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:43.580351 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:45.380987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:45.380953 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:46.153315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:46.153285 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:42:46.983385 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:46.983349 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:47.220350 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:47.220309 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:47.378953 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:47.378924 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:48.678906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:48.678863 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:50.054674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.054627 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx"] Nov 03 16:42:50.055196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.054934 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" podUID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" containerName="maas-api" containerID="cri-o://7a4315b07adfc8f2cb794bfc6c3bc119aa0bd5105277fd32e44da9ef784523f1" gracePeriod=30 Nov 03 16:42:50.182225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.182189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:50.219323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.219107 4539 generic.go:356] "Generic (PLEG): container finished" podID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" containerID="7a4315b07adfc8f2cb794bfc6c3bc119aa0bd5105277fd32e44da9ef784523f1" exitCode=0 Nov 03 16:42:50.219323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.219249 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" event={"ID":"4d4449a1-880c-47b4-bfa5-9a3caeb5f613","Type":"ContainerDied","Data":"7a4315b07adfc8f2cb794bfc6c3bc119aa0bd5105277fd32e44da9ef784523f1"} Nov 03 16:42:50.633450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.632554 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:42:50.745476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.744650 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls\") pod \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " Nov 03 16:42:50.745476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.744728 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs\") pod \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " Nov 03 16:42:50.745476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.744785 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj\" (UniqueName: \"kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj\") pod \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " Nov 03 16:42:50.745476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.744866 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca\") pod \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\" (UID: \"4d4449a1-880c-47b4-bfa5-9a3caeb5f613\") " Nov 03 16:42:50.752457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.748600 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs" pod "4d4449a1-880c-47b4-bfa5-9a3caeb5f613" (UID: "4d4449a1-880c-47b4-bfa5-9a3caeb5f613"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:42:50.752457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.748827 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca" pod "4d4449a1-880c-47b4-bfa5-9a3caeb5f613" (UID: "4d4449a1-880c-47b4-bfa5-9a3caeb5f613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:50.752457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.750866 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj" pod "4d4449a1-880c-47b4-bfa5-9a3caeb5f613" (UID: "4d4449a1-880c-47b4-bfa5-9a3caeb5f613"). InnerVolumeSpecName "kube-api-access-g67qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:50.760342 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.760299 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls" pod "4d4449a1-880c-47b4-bfa5-9a3caeb5f613" (UID: "4d4449a1-880c-47b4-bfa5-9a3caeb5f613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:42:50.847187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.847103 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:50.847187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.847133 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:50.847187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.847143 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67qj\" (UniqueName: \"kubernetes.io/projected/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-kube-api-access-g67qj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:50.847187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:50.847153 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4449a1-880c-47b4-bfa5-9a3caeb5f613-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:51.172471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.172315 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:51.233552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.233512 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" Nov 03 16:42:51.233764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.233512 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" event={"ID":"4d4449a1-880c-47b4-bfa5-9a3caeb5f613","Type":"ContainerDied","Data":"4ecbc634ecbbd6f711393d704a0d4804027f40a2cb4af120f3ac9c5563e90b27"} Nov 03 16:42:51.233764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.233684 4539 scope.go:122] "RemoveContainer" containerID="7a4315b07adfc8f2cb794bfc6c3bc119aa0bd5105277fd32e44da9ef784523f1" Nov 03 16:42:51.285756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.285718 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx"] Nov 03 16:42:51.291555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.291527 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx"] Nov 03 16:42:51.766259 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.766231 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11339-5c4f19"] Nov 03 16:42:51.766852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.766839 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" containerName="maas-api" Nov 03 16:42:51.766908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.766853 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" containerName="maas-api" Nov 03 16:42:51.766947 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.766932 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" containerName="maas-api" Nov 03 16:42:51.771516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.771496 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:51.787628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.787596 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 16:42:51.789176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.789153 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11339-5c4f19"] Nov 03 16:42:51.861814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.861769 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vxn\" (UniqueName: \"kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn\") pod \"test-curl-11339-5c4f19\" (UID: \"fd0e44ad-494b-4023-bc88-9ec47ee3e85e\") " pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:51.963332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.963272 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vxn\" (UniqueName: \"kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn\") pod \"test-curl-11339-5c4f19\" (UID: \"fd0e44ad-494b-4023-bc88-9ec47ee3e85e\") " pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:51.985654 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:51.985613 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vxn\" (UniqueName: \"kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn\") pod \"test-curl-11339-5c4f19\" (UID: \"fd0e44ad-494b-4023-bc88-9ec47ee3e85e\") " pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:52.084144 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:52.084099 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:52.394346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:52.394309 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11339-5c4f19"] Nov 03 16:42:52.395295 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:42:52.395270 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd0e44ad_494b_4023_bc88_9ec47ee3e85e.slice/crio-0bff2296736b82655cc41f0474bc6d9d1eeea745b3925edd7d2f07b9a2b3dd43 WatchSource:0}: Error finding container 0bff2296736b82655cc41f0474bc6d9d1eeea745b3925edd7d2f07b9a2b3dd43: Status 404 returned error can't find the container with id 0bff2296736b82655cc41f0474bc6d9d1eeea745b3925edd7d2f07b9a2b3dd43 Nov 03 16:42:52.469565 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:52.469525 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" path="/var/lib/kubelet/pods/4d4449a1-880c-47b4-bfa5-9a3caeb5f613/volumes" Nov 03 16:42:52.847047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:52.846977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:53.252274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:53.252179 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11339-5c4f19" event={"ID":"fd0e44ad-494b-4023-bc88-9ec47ee3e85e","Type":"ContainerStarted","Data":"ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6"} Nov 03 16:42:53.252274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:53.252230 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11339-5c4f19" event={"ID":"fd0e44ad-494b-4023-bc88-9ec47ee3e85e","Type":"ContainerStarted","Data":"0bff2296736b82655cc41f0474bc6d9d1eeea745b3925edd7d2f07b9a2b3dd43"} Nov 03 16:42:53.302104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:53.302048 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11339-5c4f19" podStartSLOduration=1.920779598 podStartE2EDuration="2.302033651s" totalImagesPullingTime="381.254053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:53.301711024 +0000 UTC m=+1716.994982046" watchObservedRunningTime="2026-11-03 16:42:53.302033651 +0000 UTC m=+1716.995304672" Nov 03 16:42:53.596115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:53.596079 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d89163-698c747997-lfjvx" podUID="4d4449a1-880c-47b4-bfa5-9a3caeb5f613" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.84:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:42:54.209837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:54.207401 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11339-5c4f19"] Nov 03 16:42:54.216033 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:54.213726 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11339-5c4f19"] Nov 03 16:42:54.448845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:54.448814 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:54.585309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:54.585274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:54.991179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:54.991103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:55.268521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:55.268413 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11339-5c4f19" podUID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" containerName="test-curl-11339-5c4f19" containerID="cri-o://ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6" gracePeriod=2 Nov 03 16:42:55.393960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:55.393909 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:55.919120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:55.919088 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:56.170750 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:56.170653 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:42:57.079589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:57.079557 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:57.492939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:57.492916 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:57.665736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:57.665648 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn\" (UniqueName: \"kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn\") pod \"fd0e44ad-494b-4023-bc88-9ec47ee3e85e\" (UID: \"fd0e44ad-494b-4023-bc88-9ec47ee3e85e\") " Nov 03 16:42:57.667911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:57.667879 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn" pod "fd0e44ad-494b-4023-bc88-9ec47ee3e85e" (UID: "fd0e44ad-494b-4023-bc88-9ec47ee3e85e"). InnerVolumeSpecName "kube-api-access-d9vxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:57.767211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:57.767181 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9vxn\" (UniqueName: \"kubernetes.io/projected/fd0e44ad-494b-4023-bc88-9ec47ee3e85e-kube-api-access-d9vxn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:42:58.292163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.292124 4539 generic.go:356] "Generic (PLEG): container finished" podID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" containerID="ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6" exitCode=137 Nov 03 16:42:58.292356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.292175 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:58.292356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.292235 4539 scope.go:122] "RemoveContainer" containerID="ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6" Nov 03 16:42:58.303229 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.303204 4539 scope.go:122] "RemoveContainer" containerID="ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6" Nov 03 16:42:58.303610 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:58.303590 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6\": container with ID starting with ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6 not found: ID does not exist" containerID="ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6" Nov 03 16:42:58.303700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.303624 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6"} err="failed to get container status \"ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6\": rpc error: code = NotFound desc = could not find container \"ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6\": container with ID starting with ff13561f2553cb846c0dcc877598debfaf4f0a4f9a6cc3454c3f1098e69ecac6 not found: ID does not exist" Nov 03 16:42:58.313335 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:42:58.313299 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11339-5c4f19\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" pod="opendatahub/test-curl-11339-5c4f19" Nov 03 16:42:58.469616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.469586 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" path="/var/lib/kubelet/pods/fd0e44ad-494b-4023-bc88-9ec47ee3e85e/volumes" Nov 03 16:42:58.671549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:58.671461 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:42:59.658758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:42:59.658724 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:00.172583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:00.172547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:00.399933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:00.399894 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:00.595307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:00.595267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:02.204952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:02.204917 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.138035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:04.138000 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.316448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:04.315963 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.518320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:04.518159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:04.685451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:04.685378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:05.212447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.211503 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.298451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.297719 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:43:05.507795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.507689 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.510060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.510026 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.533450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.522622 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:43:05.552457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.552165 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:43:05.552457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.552230 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.793768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:05.793732 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:06.102011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:06.101916 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:06.194613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:06.194581 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:06.372140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:06.372050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:06.605025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:06.604985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:06.912927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:06.912895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:07.067379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:07.067347 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:08.475198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:08.475158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:10.008407 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:10.008375 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:10.443349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:10.443306 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:11.704821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:11.704784 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:13.232930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.232840 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:13.587682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.587653 4539 generic.go:356] "Generic (PLEG): container finished" podID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerID="45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea" exitCode=0 Nov 03 16:43:13.587864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.587727 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" event={"ID":"099be21c-610d-4c5e-82ee-b3689ba56ef8","Type":"ContainerDied","Data":"45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea"} Nov 03 16:43:13.588125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.588108 4539 scope.go:122] "RemoveContainer" containerID="45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea" Nov 03 16:43:13.589556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.589530 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:43:13.589623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.589598 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:43:13.589661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:13.589627 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:43:14.598512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:14.598474 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" event={"ID":"099be21c-610d-4c5e-82ee-b3689ba56ef8","Type":"ContainerStarted","Data":"4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad"} Nov 03 16:43:14.867607 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:14.867520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:15.144024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:15.143938 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:43:15.144185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:15.144151 4539 patch_prober.go:29] interesting pod/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:43:15.144234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:15.144213 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 16:43:16.144007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.143970 4539 patch_prober.go:29] interesting pod/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:43:16.144438 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.144033 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 16:43:16.527504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.518952 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj"] Nov 03 16:43:16.527504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.519927 4539 cpu_manager.go:414] "containerMap: removing container" podUID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" containerName="test-curl-11339-5c4f19" Nov 03 16:43:16.527504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.519947 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" containerName="test-curl-11339-5c4f19" Nov 03 16:43:16.527504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.520071 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0e44ad-494b-4023-bc88-9ec47ee3e85e" containerName="test-curl-11339-5c4f19" Nov 03 16:43:16.551867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.551841 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj"] Nov 03 16:43:16.557652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.552138 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.577021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.576993 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4e3be7-serving-cert\"" Nov 03 16:43:16.638024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.637966 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:16.646696 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.646663 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.646857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.646736 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.646857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.646755 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.646857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.646806 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj268\" (UniqueName: \"kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.752011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.748269 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj268\" (UniqueName: \"kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.752659 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.752627 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.752917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.752896 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.753052 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.753035 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.754225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.754197 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.757241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.756733 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.757241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.757184 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.773066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.773032 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj268\" (UniqueName: \"kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268\") pod \"maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.794632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.785360 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:16.882331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.882301 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:16.932276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:16.932202 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:17.144640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:17.144540 4539 patch_prober.go:29] interesting pod/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:43:17.144640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:17.144606 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 16:43:17.238545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:17.238509 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj"] Nov 03 16:43:17.239218 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:43:17.239186 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf65ec_0b59_472f_abbd_21166349be99.slice/crio-347c11d3367cc8ddc04c5fcaa117ed6f258d26c038fab1fe0213919eb3dcb57b WatchSource:0}: Error finding container 347c11d3367cc8ddc04c5fcaa117ed6f258d26c038fab1fe0213919eb3dcb57b: Status 404 returned error can't find the container with id 347c11d3367cc8ddc04c5fcaa117ed6f258d26c038fab1fe0213919eb3dcb57b Nov 03 16:43:17.420372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:17.420243 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:17.676626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:17.676450 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" event={"ID":"0ddf65ec-0b59-472f-abbd-21166349be99","Type":"ContainerStarted","Data":"347c11d3367cc8ddc04c5fcaa117ed6f258d26c038fab1fe0213919eb3dcb57b"} Nov 03 16:43:18.088465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.086378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:18.152508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.151922 4539 patch_prober.go:29] interesting pod/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:43:18.152508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.151988 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 16:43:18.294165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.294131 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:18.686788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.686747 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" event={"ID":"0ddf65ec-0b59-472f-abbd-21166349be99","Type":"ContainerStarted","Data":"c963412b07d22cc49e7c06b59eaa5385d436233634b5c6f87f8fd9fdf1572cb9"} Nov 03 16:43:18.687007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.686836 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:18.739249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.739194 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" podStartSLOduration=2.409136931 podStartE2EDuration="2.739178768s" totalImagesPullingTime="330.041837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:18.734852655 +0000 UTC m=+1742.428123678" watchObservedRunningTime="2026-11-03 16:43:18.739178768 +0000 UTC m=+1742.432449790" Nov 03 16:43:18.789882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:18.789853 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:19.143577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:19.143544 4539 patch_prober.go:29] interesting pod/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:43:19.143790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:19.143598 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 16:43:20.148588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:20.148560 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:43:20.149018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:20.148997 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:43:20.149847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:20.149832 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:43:20.535294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:20.535264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:21.544055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:21.544020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:21.713372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:21.713334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:22.355697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:22.355655 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:22.839580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:22.839531 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:23.264515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:23.264403 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:23.406680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:23.406634 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:24.176622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:24.176587 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:24.698890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:24.698865 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:43:25.881920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:25.881880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:26.161076 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:43:26.160993 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:43:27.501696 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:27.501629 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:29.868504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:29.867898 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:32.267055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:32.267021 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:34.146109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:34.146069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:35.571137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:35.571101 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk"] Nov 03 16:43:35.572032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:35.571980 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" podUID="2b6cb3eb-7451-4f94-aac1-d3d6f712d468" containerName="maas-api" containerID="cri-o://eac57e304eacd61f400a2f34cb00955a2affbeb6ca91e557e83f23c548ad2978" gracePeriod=30 Nov 03 16:43:35.664608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:35.664576 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:36.067448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.058851 4539 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb3eb-7451-4f94-aac1-d3d6f712d468" containerID="eac57e304eacd61f400a2f34cb00955a2affbeb6ca91e557e83f23c548ad2978" exitCode=0 Nov 03 16:43:36.067851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.063316 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" event={"ID":"2b6cb3eb-7451-4f94-aac1-d3d6f712d468","Type":"ContainerDied","Data":"eac57e304eacd61f400a2f34cb00955a2affbeb6ca91e557e83f23c548ad2978"} Nov 03 16:43:36.237358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.237331 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:43:36.302476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.300546 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52\" (UniqueName: \"kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52\") pod \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " Nov 03 16:43:36.302476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.300679 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca\") pod \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " Nov 03 16:43:36.302476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.300839 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs\") pod \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " Nov 03 16:43:36.305081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.305053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:36.318666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.318592 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca" pod "2b6cb3eb-7451-4f94-aac1-d3d6f712d468" (UID: "2b6cb3eb-7451-4f94-aac1-d3d6f712d468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:36.321737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.321699 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs" pod "2b6cb3eb-7451-4f94-aac1-d3d6f712d468" (UID: "2b6cb3eb-7451-4f94-aac1-d3d6f712d468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:43:36.328359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.328315 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52" pod "2b6cb3eb-7451-4f94-aac1-d3d6f712d468" (UID: "2b6cb3eb-7451-4f94-aac1-d3d6f712d468"). InnerVolumeSpecName "kube-api-access-nph52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:36.404479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.404411 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls\") pod \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\" (UID: \"2b6cb3eb-7451-4f94-aac1-d3d6f712d468\") " Nov 03 16:43:36.413031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.412985 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh"] Nov 03 16:43:36.414238 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.414177 4539 cpu_manager.go:414] "containerMap: removing container" podUID="2b6cb3eb-7451-4f94-aac1-d3d6f712d468" containerName="maas-api" Nov 03 16:43:36.414416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.414399 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6cb3eb-7451-4f94-aac1-d3d6f712d468" containerName="maas-api" Nov 03 16:43:36.414730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.414713 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6cb3eb-7451-4f94-aac1-d3d6f712d468" containerName="maas-api" Nov 03 16:43:36.418755 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.418726 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nph52\" (UniqueName: \"kubernetes.io/projected/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-kube-api-access-nph52\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:43:36.418755 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.418755 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:43:36.418949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.418769 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:43:36.422335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.422318 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.439890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.438974 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls" pod "2b6cb3eb-7451-4f94-aac1-d3d6f712d468" (UID: "2b6cb3eb-7451-4f94-aac1-d3d6f712d468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:43:36.439890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.439214 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bffe15-serving-cert\"" Nov 03 16:43:36.526151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.526112 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.526416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.526399 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.526692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.526676 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmtc\" (UniqueName: \"kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.526908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.526894 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.527141 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.527130 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cb3eb-7451-4f94-aac1-d3d6f712d468-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:43:36.600311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.600132 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh"] Nov 03 16:43:36.616710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.615913 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.637601 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.637654 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.637736 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmtc\" (UniqueName: \"kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.637828 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:43:36.637939 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-serving-cert: secret "maas-api-e2e-api-a-bffe15-serving-cert" not found Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:43:36.638006 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls podName:8f531f1d-6b44-4499-b1c2-cac8a28ef78d nodeName:}" failed. No retries permitted until 2026-11-03 16:43:37.137987252 +0000 UTC m=+1760.831258268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls") pod "maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" (UID: "8f531f1d-6b44-4499-b1c2-cac8a28ef78d") : secret "maas-api-e2e-api-a-bffe15-serving-cert" not found Nov 03 16:43:36.641733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.638669 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.666509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.660328 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.674446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.672847 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmtc\" (UniqueName: \"kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:36.766674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.766639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:36.920918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:36.920818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:37.076390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.076360 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" Nov 03 16:43:37.076604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.076401 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk" event={"ID":"2b6cb3eb-7451-4f94-aac1-d3d6f712d468","Type":"ContainerDied","Data":"b54de4d2a942f28203e99050bd33aac344f1063b65dcea43aa7fc5a9a367792d"} Nov 03 16:43:37.076604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.076485 4539 scope.go:122] "RemoveContainer" containerID="eac57e304eacd61f400a2f34cb00955a2affbeb6ca91e557e83f23c548ad2978" Nov 03 16:43:37.127588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.127554 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:37.131963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.131921 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk"] Nov 03 16:43:37.138018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.137991 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-k6xqk"] Nov 03 16:43:37.151396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.148635 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:37.152770 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.152740 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\") pod \"maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:37.392172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.392138 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:37.747530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:37.747349 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh"] Nov 03 16:43:37.769032 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:43:37.768993 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f531f1d_6b44_4499_b1c2_cac8a28ef78d.slice/crio-970b89c7fc9ecc06b2c418afdc81d671604031b5786088eaa483c0b1d008e580 WatchSource:0}: Error finding container 970b89c7fc9ecc06b2c418afdc81d671604031b5786088eaa483c0b1d008e580: Status 404 returned error can't find the container with id 970b89c7fc9ecc06b2c418afdc81d671604031b5786088eaa483c0b1d008e580 Nov 03 16:43:38.085218 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:38.085154 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" event={"ID":"8f531f1d-6b44-4499-b1c2-cac8a28ef78d","Type":"ContainerStarted","Data":"970b89c7fc9ecc06b2c418afdc81d671604031b5786088eaa483c0b1d008e580"} Nov 03 16:43:38.558459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:38.551621 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6cb3eb-7451-4f94-aac1-d3d6f712d468" path="/var/lib/kubelet/pods/2b6cb3eb-7451-4f94-aac1-d3d6f712d468/volumes" Nov 03 16:43:38.864457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:38.864341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:39.095492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:39.095410 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" event={"ID":"8f531f1d-6b44-4499-b1c2-cac8a28ef78d","Type":"ContainerStarted","Data":"27bb74f1fd9f8560f364cdea1c1e32f2f66afee531c49d535437002d928ca389"} Nov 03 16:43:39.095672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:39.095539 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:39.140503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:39.139559 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" podStartSLOduration=2.804210928 podStartE2EDuration="3.139541494s" totalImagesPullingTime="335.330566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:39.137988544 +0000 UTC m=+1762.831259567" watchObservedRunningTime="2026-11-03 16:43:39.139541494 +0000 UTC m=+1762.832812518" Nov 03 16:43:40.470824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:40.470794 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:40.687930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:40.687891 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:42.193438 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:42.193369 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:43.394449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:43.394399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:43.548448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:43.548281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:44.032446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:44.032112 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:44.251560 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:44.251530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:44.460099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:44.460003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:44.605344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:44.605301 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:45.114137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:45.114111 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:43:45.207513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:45.207479 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:45.376831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:45.376741 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:45.981768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:45.981735 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc"] Nov 03 16:43:46.029693 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.029659 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc"] Nov 03 16:43:46.030097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.030011 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.056935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.056908 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-156c2a-openshift-default-dockercfg-l4dr7\"" Nov 03 16:43:46.109364 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.109324 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.109653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.109629 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.109879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.109859 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.110010 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.109997 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.110175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.110160 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.110276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.110262 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.110383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.110368 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.110539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.110522 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.110646 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.110632 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqc6\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.141524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.139913 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.211948 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212017 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212095 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212126 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212220 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212243 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212281 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212318 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.212448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212348 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqc6\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.213066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.212830 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.213124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.213066 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.214493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.213788 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.214493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.213851 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.214493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.214054 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.218331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.218237 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.220608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.220588 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.237051 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.236960 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqc6\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.238651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.238626 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token\") pod \"e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.432734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.432698 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:46.455363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.455329 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:46.824978 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.824942 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc"] Nov 03 16:43:46.825178 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:43:46.825149 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c42f87_5b3f_4724_81b5_18be65528ec0.slice/crio-e385a227431ea661b41aefd387880f1d5709d4d75420bb485449116f31c34f0f WatchSource:0}: Error finding container e385a227431ea661b41aefd387880f1d5709d4d75420bb485449116f31c34f0f: Status 404 returned error can't find the container with id e385a227431ea661b41aefd387880f1d5709d4d75420bb485449116f31c34f0f Nov 03 16:43:46.828201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.828163 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:43:46.828327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.828235 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:43:46.828327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:46.828270 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:43:47.017932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:47.017894 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:47.185818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:47.185738 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" event={"ID":"a7c42f87-5b3f-4724-81b5-18be65528ec0","Type":"ContainerStarted","Data":"c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f"} Nov 03 16:43:47.185818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:47.185791 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" event={"ID":"a7c42f87-5b3f-4724-81b5-18be65528ec0","Type":"ContainerStarted","Data":"e385a227431ea661b41aefd387880f1d5709d4d75420bb485449116f31c34f0f"} Nov 03 16:43:47.240896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:47.240842 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podStartSLOduration=2.240826209 podStartE2EDuration="2.240826209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:47.235515654 +0000 UTC m=+1770.928786675" watchObservedRunningTime="2026-11-03 16:43:47.240826209 +0000 UTC m=+1770.934097231" Nov 03 16:43:47.456552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:47.456224 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:47.476794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:47.476758 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:48.192317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:48.192284 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:48.193292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:48.193274 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:43:49.078372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:49.078278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:49.754584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:49.754552 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:49.954036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:49.954007 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:50.983321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:50.983290 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:51.185694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:51.185663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:51.680946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:51.680911 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:51.839317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:51.839283 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:52.270772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:52.270739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:52.888456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:52.888395 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:54.970582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:54.970546 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:55.120526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:55.120494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:55.277986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:55.277954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:56.165156 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:43:56.165094 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:43:57.103724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:57.103684 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:43:59.221392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:43:59.221355 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:00.597095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:00.596980 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2"] Nov 03 16:44:00.597632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:00.597321 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" podUID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" containerName="maas-api" containerID="cri-o://ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d" gracePeriod=30 Nov 03 16:44:00.702716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:00.702673 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:01.247449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.242147 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:44:01.317633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.315283 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d\" (UniqueName: \"kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d\") pod \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " Nov 03 16:44:01.317633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.315391 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs\") pod \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " Nov 03 16:44:01.317633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.315580 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca\") pod \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " Nov 03 16:44:01.317633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.315630 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\") pod \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\" (UID: \"9fddbed9-6b75-4677-a72f-2a1864e6ec7b\") " Nov 03 16:44:01.325446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.322903 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls" pod "9fddbed9-6b75-4677-a72f-2a1864e6ec7b" (UID: "9fddbed9-6b75-4677-a72f-2a1864e6ec7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:01.326731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.325921 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d" pod "9fddbed9-6b75-4677-a72f-2a1864e6ec7b" (UID: "9fddbed9-6b75-4677-a72f-2a1864e6ec7b"). InnerVolumeSpecName "kube-api-access-sx95d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:01.326731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.326274 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca" pod "9fddbed9-6b75-4677-a72f-2a1864e6ec7b" (UID: "9fddbed9-6b75-4677-a72f-2a1864e6ec7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:01.338558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.338503 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs" pod "9fddbed9-6b75-4677-a72f-2a1864e6ec7b" (UID: "9fddbed9-6b75-4677-a72f-2a1864e6ec7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:01.352522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.351346 4539 generic.go:356] "Generic (PLEG): container finished" podID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" containerID="ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d" exitCode=0 Nov 03 16:44:01.352522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.351403 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" Nov 03 16:44:01.352522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.351640 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" event={"ID":"9fddbed9-6b75-4677-a72f-2a1864e6ec7b","Type":"ContainerDied","Data":"ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d"} Nov 03 16:44:01.352522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.351693 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" event={"ID":"9fddbed9-6b75-4677-a72f-2a1864e6ec7b","Type":"ContainerDied","Data":"cbcd9b87a1d1325fa8f4b41ab55643983209d52c555f7094221b18ea19066968"} Nov 03 16:44:01.352522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.351721 4539 scope.go:122] "RemoveContainer" containerID="ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d" Nov 03 16:44:01.369868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.369842 4539 scope.go:122] "RemoveContainer" containerID="ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d" Nov 03 16:44:01.370213 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:01.370192 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d\": container with ID starting with ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d not found: ID does not exist" containerID="ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d" Nov 03 16:44:01.370276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.370225 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d"} err="failed to get container status \"ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d\": rpc error: code = NotFound desc = could not find container \"ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d\": container with ID starting with ed6230cc19aafa08a23e1ade8205ef72d04685d2ad5400fe92fadc837deda02d not found: ID does not exist" Nov 03 16:44:01.402482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.402443 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2"] Nov 03 16:44:01.428234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.428204 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:01.428545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.428244 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:01.428545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.428260 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:01.428545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.428279 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx95d\" (UniqueName: \"kubernetes.io/projected/9fddbed9-6b75-4677-a72f-2a1864e6ec7b-kube-api-access-sx95d\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:01.431910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.431876 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2"] Nov 03 16:44:01.623413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:01.623370 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:02.377783 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.377748 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:02.470017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.469984 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" path="/var/lib/kubelet/pods/9fddbed9-6b75-4677-a72f-2a1864e6ec7b/volumes" Nov 03 16:44:02.773347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.773316 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz"] Nov 03 16:44:02.773979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.773963 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" containerName="maas-api" Nov 03 16:44:02.774056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.773981 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" containerName="maas-api" Nov 03 16:44:02.774129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.774118 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" containerName="maas-api" Nov 03 16:44:02.782084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.780279 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.820179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.820124 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz"] Nov 03 16:44:02.824746 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.824720 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-db3c6b-openshift-default-dockercfg-z4bcr\"" Nov 03 16:44:02.863217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863169 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863304 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863355 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gms2b\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863464 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863512 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863566 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863607 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863658 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.863917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.863703 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.865881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.865853 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:02.967448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.966506 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.967448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.966590 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gms2b\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.967448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.966705 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.967448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.966775 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.977539 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.977608 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.977670 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.977721 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.977853 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.978388 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.978700 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.979280 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.980942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.979985 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:02.995186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.994737 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:03.002450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:02.995591 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:03.041496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.035953 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:03.054454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.052989 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:03.068451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.061140 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gms2b\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:03.073821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.069914 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token\") pod \"e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:03.157456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.156297 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:03.389247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.388892 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:03.562969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.562937 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz"] Nov 03 16:44:03.568585 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:44:03.568549 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cbb72b_92ba_4c9c_bc76_ceb86906a0b5.slice/crio-3eb3be91d504c8f103be4b6b6df1a9a1b76df1b11ecb515f52f4daf46f49f275 WatchSource:0}: Error finding container 3eb3be91d504c8f103be4b6b6df1a9a1b76df1b11ecb515f52f4daf46f49f275: Status 404 returned error can't find the container with id 3eb3be91d504c8f103be4b6b6df1a9a1b76df1b11ecb515f52f4daf46f49f275 Nov 03 16:44:03.571598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.571556 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:03.571715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.571650 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:03.571715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.571698 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:03.599067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.599033 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:03.697013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.695840 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6"] Nov 03 16:44:03.708667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.702836 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.743346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.743303 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-73cc79-openshift-default-dockercfg-8kjvv\"" Nov 03 16:44:03.754473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.754443 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6"] Nov 03 16:44:03.836014 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.835972 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:03.851091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851053 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851127 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851201 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851300 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851366 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851462 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851568 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqg8n\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851676 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.851863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.851824 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.954831 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqg8n\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.954939 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955065 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955116 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955189 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955262 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955374 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955445 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.955537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.955521 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.956216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.956101 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.956802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.956768 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.957863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.957079 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.957863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.957394 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.961394 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.961349 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.970491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.966413 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.970491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.967017 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:03.999614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:03.999563 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqg8n\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:04.008866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.007220 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token\") pod \"e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:04.059561 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.059524 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:04.109521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.109451 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:04.151851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.151812 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6fa98e-74499d6898-85px2" podUID="9fddbed9-6b75-4677-a72f-2a1864e6ec7b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.86:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:44:04.380368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.380322 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" event={"ID":"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5","Type":"ContainerStarted","Data":"7d3e69ab48de4e7f90a6d5625e5a6a86eec737e5b726ec72db3c7401ec26f98b"} Nov 03 16:44:04.380602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.380397 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" event={"ID":"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5","Type":"ContainerStarted","Data":"3eb3be91d504c8f103be4b6b6df1a9a1b76df1b11ecb515f52f4daf46f49f275"} Nov 03 16:44:04.404234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.404204 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6"] Nov 03 16:44:04.404389 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:44:04.404278 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131eb6da_b0bf_49df_9a3f_33480f237df0.slice/crio-eeb0d8a4380a55e37bef59ae2646b8b875b6e68e4f12d78de29ee742006796da WatchSource:0}: Error finding container eeb0d8a4380a55e37bef59ae2646b8b875b6e68e4f12d78de29ee742006796da: Status 404 returned error can't find the container with id eeb0d8a4380a55e37bef59ae2646b8b875b6e68e4f12d78de29ee742006796da Nov 03 16:44:04.407155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.407122 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:04.407247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.407199 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:04.407247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.407240 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:04.431070 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:04.428496 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podStartSLOduration=2.428477641 podStartE2EDuration="2.428477641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:04.426879096 +0000 UTC m=+1788.120150118" watchObservedRunningTime="2026-11-03 16:44:04.428477641 +0000 UTC m=+1788.121748665" Nov 03 16:44:05.158637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.158602 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:05.164283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.164254 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:05.392083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.391975 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:05.393087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.393054 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" event={"ID":"131eb6da-b0bf-49df-9a3f-33480f237df0","Type":"ContainerStarted","Data":"f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248"} Nov 03 16:44:05.393294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.393277 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" event={"ID":"131eb6da-b0bf-49df-9a3f-33480f237df0","Type":"ContainerStarted","Data":"eeb0d8a4380a55e37bef59ae2646b8b875b6e68e4f12d78de29ee742006796da"} Nov 03 16:44:05.393804 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.393783 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:05.395596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.395577 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:44:05.439542 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:05.438025 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podStartSLOduration=2.438005771 podStartE2EDuration="2.438005771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:05.436537954 +0000 UTC m=+1789.129809004" watchObservedRunningTime="2026-11-03 16:44:05.438005771 +0000 UTC m=+1789.131276819" Nov 03 16:44:06.060608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:06.060576 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:06.066884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:06.066856 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:06.400806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:06.400698 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:06.402300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:06.402273 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:44:06.840172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:06.840137 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:07.001207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:07.001174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:07.164148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:07.164069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:07.369310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:07.369268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:08.185586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:08.185550 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:08.643474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:08.643393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:08.922117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:08.921996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:09.179925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:09.179798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:09.744484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:09.744447 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:09.895334 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:09.895295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:10.112145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:10.112115 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss"] Nov 03 16:44:10.112681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:10.112648 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" containerID="cri-o://4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad" gracePeriod=30 Nov 03 16:44:10.159538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:10.159498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:10.649870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:10.649841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:10.805185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:10.805150 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:11.023653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:11.023619 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:12.041041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:12.041004 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:12.517853 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:12.517772 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:12.711096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:12.711064 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:14.176346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:14.176267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:15.411712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.411683 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:44:15.518226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.518191 4539 generic.go:356] "Generic (PLEG): container finished" podID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerID="4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad" exitCode=0 Nov 03 16:44:15.518465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.518241 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" event={"ID":"099be21c-610d-4c5e-82ee-b3689ba56ef8","Type":"ContainerDied","Data":"4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad"} Nov 03 16:44:15.518465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.518260 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" Nov 03 16:44:15.518465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.518295 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss" event={"ID":"099be21c-610d-4c5e-82ee-b3689ba56ef8","Type":"ContainerDied","Data":"f8ac81a3701770d8c1d47218f38aaeeb97bffeef5d25a993d8463ecc835ae062"} Nov 03 16:44:15.518465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.518324 4539 scope.go:122] "RemoveContainer" containerID="4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad" Nov 03 16:44:15.537802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.537771 4539 scope.go:122] "RemoveContainer" containerID="45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea" Nov 03 16:44:15.549388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.549365 4539 scope.go:122] "RemoveContainer" containerID="4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad" Nov 03 16:44:15.549701 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:15.549679 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad\": container with ID starting with 4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad not found: ID does not exist" containerID="4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad" Nov 03 16:44:15.549795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.549712 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad"} err="failed to get container status \"4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad\": rpc error: code = NotFound desc = could not find container \"4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad\": container with ID starting with 4c577be4fab12e55faaf27609bf8c67b9f9c715a74654d0f17a06f5fe4b5c1ad not found: ID does not exist" Nov 03 16:44:15.549795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.549736 4539 scope.go:122] "RemoveContainer" containerID="45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea" Nov 03 16:44:15.550012 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:15.549995 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea\": container with ID starting with 45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea not found: ID does not exist" containerID="45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea" Nov 03 16:44:15.550054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.550018 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea"} err="failed to get container status \"45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea\": rpc error: code = NotFound desc = could not find container \"45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea\": container with ID starting with 45b0cd2f44c1364d78ff880dc1177d8a5c4d8e00b9accc51bc947015931b96ea not found: ID does not exist" Nov 03 16:44:15.589657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589629 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.589837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589708 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.589837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589771 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.589837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589820 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.590009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589845 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.590009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589876 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.590009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.589960 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.590157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.590055 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.590157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.590123 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data\") pod \"099be21c-610d-4c5e-82ee-b3689ba56ef8\" (UID: \"099be21c-610d-4c5e-82ee-b3689ba56ef8\") " Nov 03 16:44:15.590157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.590119 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:15.590304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.590205 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:15.591686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.590657 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:15.591686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.590742 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:15.591686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.591381 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.591686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.591401 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099be21c-610d-4c5e-82ee-b3689ba56ef8-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.591686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.591415 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.591686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.591444 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.592103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.591944 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:15.592506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.592486 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:15.592597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.592524 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:15.592672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.592659 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "kube-api-access-wkdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:15.592833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.592820 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo" pod "099be21c-610d-4c5e-82ee-b3689ba56ef8" (UID: "099be21c-610d-4c5e-82ee-b3689ba56ef8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:44:15.693026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.692950 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.693026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.692978 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.693026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.692988 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.693026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.692998 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdzk\" (UniqueName: \"kubernetes.io/projected/099be21c-610d-4c5e-82ee-b3689ba56ef8-kube-api-access-wkdzk\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.693026 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.693007 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099be21c-610d-4c5e-82ee-b3689ba56ef8-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:15.854730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.854696 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:15.864553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.864526 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss"] Nov 03 16:44:15.875381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:15.875345 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6fa98e-openshift-default-5c9457fb7f-wdmss"] Nov 03 16:44:16.469215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:16.469182 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" path="/var/lib/kubelet/pods/099be21c-610d-4c5e-82ee-b3689ba56ef8/volumes" Nov 03 16:44:16.744846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:16.744761 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:44:16.746186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:16.746163 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:44:16.768304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:16.768271 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:16.857292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:16.857263 4539 scope.go:122] "RemoveContainer" containerID="7e57cc6df6381f5d3592b2e2e170a7125f2619e984233361b6c103934e6c841b" Nov 03 16:44:17.510809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:17.510778 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:17.735448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:17.735234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:17.905784 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:17.905755 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:18.050536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:18.050505 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:19.692847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:19.692814 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:21.377833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:21.377797 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:23.054852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.054818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:23.260045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.260014 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11339-80937f"] Nov 03 16:44:23.260625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.260611 4539 cpu_manager.go:414] "containerMap: removing container" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" Nov 03 16:44:23.260674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.260626 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" Nov 03 16:44:23.260735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.260726 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" Nov 03 16:44:23.260772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.260736 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" Nov 03 16:44:23.265563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.265544 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:23.285609 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.285581 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11339-80937f"] Nov 03 16:44:23.289944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.289916 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 16:44:23.312012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.311916 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwj5\" (UniqueName: \"kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5\") pod \"test-curl-11339-80937f\" (UID: \"dfe03662-4898-4502-ad40-853ec9babbd9\") " pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:23.413341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.413310 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwj5\" (UniqueName: \"kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5\") pod \"test-curl-11339-80937f\" (UID: \"dfe03662-4898-4502-ad40-853ec9babbd9\") " pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:23.437065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.437034 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwj5\" (UniqueName: \"kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5\") pod \"test-curl-11339-80937f\" (UID: \"dfe03662-4898-4502-ad40-853ec9babbd9\") " pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:23.578080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.577992 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:23.882451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:23.882375 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11339-80937f"] Nov 03 16:44:23.882560 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:44:23.882485 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfe03662_4898_4502_ad40_853ec9babbd9.slice/crio-20104bd55dfad9d2385db9aeaf359efa00bc57e754af286f933981b7c4dfbec1 WatchSource:0}: Error finding container 20104bd55dfad9d2385db9aeaf359efa00bc57e754af286f933981b7c4dfbec1: Status 404 returned error can't find the container with id 20104bd55dfad9d2385db9aeaf359efa00bc57e754af286f933981b7c4dfbec1 Nov 03 16:44:24.587110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:24.587076 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11339-80937f" event={"ID":"dfe03662-4898-4502-ad40-853ec9babbd9","Type":"ContainerStarted","Data":"200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241"} Nov 03 16:44:24.587643 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:24.587154 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11339-80937f" event={"ID":"dfe03662-4898-4502-ad40-853ec9babbd9","Type":"ContainerStarted","Data":"20104bd55dfad9d2385db9aeaf359efa00bc57e754af286f933981b7c4dfbec1"} Nov 03 16:44:24.605817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:24.605777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:24.634324 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:24.630908 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11339-80937f" podStartSLOduration=1.630883197 podStartE2EDuration="1.630883197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:24.625295338 +0000 UTC m=+1808.318566361" watchObservedRunningTime="2026-11-03 16:44:24.630883197 +0000 UTC m=+1808.324154225" Nov 03 16:44:24.825571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:24.825534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:25.491151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:25.491116 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11339-80937f"] Nov 03 16:44:25.495252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:25.495218 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11339-80937f"] Nov 03 16:44:26.071898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.071854 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:26.159227 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:26.159181 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:44:26.167019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.166987 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589"] Nov 03 16:44:26.167947 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.167928 4539 cpu_manager.go:414] "containerMap: removing container" podUID="dfe03662-4898-4502-ad40-853ec9babbd9" containerName="test-curl-11339-80937f" Nov 03 16:44:26.168028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.167951 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe03662-4898-4502-ad40-853ec9babbd9" containerName="test-curl-11339-80937f" Nov 03 16:44:26.168028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.167975 4539 cpu_manager.go:414] "containerMap: removing container" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" Nov 03 16:44:26.168028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.167982 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099be21c-610d-4c5e-82ee-b3689ba56ef8" containerName="istio-proxy" Nov 03 16:44:26.168156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.168144 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe03662-4898-4502-ad40-853ec9babbd9" containerName="test-curl-11339-80937f" Nov 03 16:44:26.172036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.172013 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.189510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.189478 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-db3c6b-serving-cert\"" Nov 03 16:44:26.214062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.214031 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589"] Nov 03 16:44:26.273553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.273512 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49np\" (UniqueName: \"kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.273731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.273580 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.273789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.273722 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.273789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.273760 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.286348 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.286283 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:26.375247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.375162 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49np\" (UniqueName: \"kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.375247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.375242 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.375503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.375331 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.375503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.375366 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.376028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.376001 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.377910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.377890 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.377966 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.377944 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.398139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.397929 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49np\" (UniqueName: \"kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np\") pod \"maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.447951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.444945 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:26.488990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.488693 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:26.635140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.629176 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11339-80937f" podUID="dfe03662-4898-4502-ad40-853ec9babbd9" containerName="test-curl-11339-80937f" containerID="cri-o://200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241" gracePeriod=2 Nov 03 16:44:26.956832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:26.956779 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589"] Nov 03 16:44:26.957136 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:44:26.957102 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7d6160_b41b_400b_9316_35478bbe98ce.slice/crio-befdb2bb4d5d935c26cc78e4aa796cceace4ba64dfc5c1638c8e51e40a8026b0 WatchSource:0}: Error finding container befdb2bb4d5d935c26cc78e4aa796cceace4ba64dfc5c1638c8e51e40a8026b0: Status 404 returned error can't find the container with id befdb2bb4d5d935c26cc78e4aa796cceace4ba64dfc5c1638c8e51e40a8026b0 Nov 03 16:44:27.654503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:27.653920 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" event={"ID":"fb7d6160-b41b-400b-9316-35478bbe98ce","Type":"ContainerStarted","Data":"befdb2bb4d5d935c26cc78e4aa796cceace4ba64dfc5c1638c8e51e40a8026b0"} Nov 03 16:44:28.065382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:28.065344 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:28.254446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:28.254126 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:28.663683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:28.663649 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" event={"ID":"fb7d6160-b41b-400b-9316-35478bbe98ce","Type":"ContainerStarted","Data":"c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6"} Nov 03 16:44:28.664072 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:28.663709 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:28.718120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:28.718047 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" podStartSLOduration=2.419080984 podStartE2EDuration="2.718025527s" totalImagesPullingTime="298.944543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:28.717893361 +0000 UTC m=+1812.411164385" watchObservedRunningTime="2026-11-03 16:44:28.718025527 +0000 UTC m=+1812.411296550" Nov 03 16:44:29.338679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.338649 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:29.423449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.423076 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5\" (UniqueName: \"kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5\") pod \"dfe03662-4898-4502-ad40-853ec9babbd9\" (UID: \"dfe03662-4898-4502-ad40-853ec9babbd9\") " Nov 03 16:44:29.436701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.436652 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5" pod "dfe03662-4898-4502-ad40-853ec9babbd9" (UID: "dfe03662-4898-4502-ad40-853ec9babbd9"). InnerVolumeSpecName "kube-api-access-zhwj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:29.541457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.541398 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhwj5\" (UniqueName: \"kubernetes.io/projected/dfe03662-4898-4502-ad40-853ec9babbd9-kube-api-access-zhwj5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:29.672268 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.672232 4539 generic.go:356] "Generic (PLEG): container finished" podID="dfe03662-4898-4502-ad40-853ec9babbd9" containerID="200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241" exitCode=137 Nov 03 16:44:29.672742 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.672282 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:29.672742 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.672335 4539 scope.go:122] "RemoveContainer" containerID="200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241" Nov 03 16:44:29.683003 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.682984 4539 scope.go:122] "RemoveContainer" containerID="200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241" Nov 03 16:44:29.683327 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:29.683307 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241\": container with ID starting with 200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241 not found: ID does not exist" containerID="200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241" Nov 03 16:44:29.683411 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.683338 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241"} err="failed to get container status \"200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241\": rpc error: code = NotFound desc = could not find container \"200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241\": container with ID starting with 200feaa1d005bbed5f547926d5bf61bbdf75d19b4a36a421784d03371cbd9241 not found: ID does not exist" Nov 03 16:44:29.692481 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:29.692447 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11339-80937f\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="dfe03662-4898-4502-ad40-853ec9babbd9" pod="opendatahub/test-curl-11339-80937f" Nov 03 16:44:29.891443 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:29.891297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:30.469081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:30.469045 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe03662-4898-4502-ad40-853ec9babbd9" path="/var/lib/kubelet/pods/dfe03662-4898-4502-ad40-853ec9babbd9/volumes" Nov 03 16:44:31.120449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:31.115589 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:31.886936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:31.886907 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:33.234325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:33.234288 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj"] Nov 03 16:44:33.234985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:33.234592 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" podUID="0ddf65ec-0b59-472f-abbd-21166349be99" containerName="maas-api" containerID="cri-o://c963412b07d22cc49e7c06b59eaa5385d436233634b5c6f87f8fd9fdf1572cb9" gracePeriod=30 Nov 03 16:44:33.338197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:33.338127 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:33.818590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:33.818406 4539 generic.go:356] "Generic (PLEG): container finished" podID="0ddf65ec-0b59-472f-abbd-21166349be99" containerID="c963412b07d22cc49e7c06b59eaa5385d436233634b5c6f87f8fd9fdf1572cb9" exitCode=0 Nov 03 16:44:33.818590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:33.818514 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" event={"ID":"0ddf65ec-0b59-472f-abbd-21166349be99","Type":"ContainerDied","Data":"c963412b07d22cc49e7c06b59eaa5385d436233634b5c6f87f8fd9fdf1572cb9"} Nov 03 16:44:33.948452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:33.939718 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:44:34.101447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.101304 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs\") pod \"0ddf65ec-0b59-472f-abbd-21166349be99\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " Nov 03 16:44:34.101873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.101786 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls\") pod \"0ddf65ec-0b59-472f-abbd-21166349be99\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " Nov 03 16:44:34.102117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.102100 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268\" (UniqueName: \"kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268\") pod \"0ddf65ec-0b59-472f-abbd-21166349be99\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " Nov 03 16:44:34.102326 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.102310 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca\") pod \"0ddf65ec-0b59-472f-abbd-21166349be99\" (UID: \"0ddf65ec-0b59-472f-abbd-21166349be99\") " Nov 03 16:44:34.104861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.103417 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca" pod "0ddf65ec-0b59-472f-abbd-21166349be99" (UID: "0ddf65ec-0b59-472f-abbd-21166349be99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:34.105604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.105585 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf65ec-0b59-472f-abbd-21166349be99-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:34.106110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.106073 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268" pod "0ddf65ec-0b59-472f-abbd-21166349be99" (UID: "0ddf65ec-0b59-472f-abbd-21166349be99"). InnerVolumeSpecName "kube-api-access-tj268". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:34.107341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.107314 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls" pod "0ddf65ec-0b59-472f-abbd-21166349be99" (UID: "0ddf65ec-0b59-472f-abbd-21166349be99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:34.107597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.107574 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs" pod "0ddf65ec-0b59-472f-abbd-21166349be99" (UID: "0ddf65ec-0b59-472f-abbd-21166349be99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:34.207756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.207714 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:34.207756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.207748 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf65ec-0b59-472f-abbd-21166349be99-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:34.207756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.207761 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj268\" (UniqueName: \"kubernetes.io/projected/0ddf65ec-0b59-472f-abbd-21166349be99-kube-api-access-tj268\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:44:34.235016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.234980 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:44:34.235412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.235086 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:44:34.279925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.279881 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:34.679632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.679605 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:44:34.828539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.828511 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" Nov 03 16:44:34.828539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.828514 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj" event={"ID":"0ddf65ec-0b59-472f-abbd-21166349be99","Type":"ContainerDied","Data":"347c11d3367cc8ddc04c5fcaa117ed6f258d26c038fab1fe0213919eb3dcb57b"} Nov 03 16:44:34.828821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.828585 4539 scope.go:122] "RemoveContainer" containerID="c963412b07d22cc49e7c06b59eaa5385d436233634b5c6f87f8fd9fdf1572cb9" Nov 03 16:44:34.879744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.879653 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj"] Nov 03 16:44:34.883943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:34.883910 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e3be7-756b9b6cfd-c7slj"] Nov 03 16:44:35.408563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.408528 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9"] Nov 03 16:44:35.409123 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.409110 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0ddf65ec-0b59-472f-abbd-21166349be99" containerName="maas-api" Nov 03 16:44:35.409168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.409125 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddf65ec-0b59-472f-abbd-21166349be99" containerName="maas-api" Nov 03 16:44:35.409257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.409248 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddf65ec-0b59-472f-abbd-21166349be99" containerName="maas-api" Nov 03 16:44:35.414801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.414781 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.426558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.426526 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.426730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.426602 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6vg\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.427400 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.427374 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.427543 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.427524 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.427612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.427565 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.427664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.427608 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.427746 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.427723 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.438807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.438779 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 16:44:35.439008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.438967 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 16:44:35.439095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.439029 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xzr7j\"" Nov 03 16:44:35.442759 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.442730 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9"] Nov 03 16:44:35.446820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.446795 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529000 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529067 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6vg\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529203 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529254 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529277 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529304 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529327 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.529941 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.535916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.533081 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.544080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.542518 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.544080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.542618 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.544363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.544339 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.553635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.553592 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6vg\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.562040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.561994 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k6pb9\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.578778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.578738 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:35.743028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.741560 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:35.743028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:35.742371 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:36.200063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:36.199951 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9"] Nov 03 16:44:36.200486 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:44:36.200417 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3292ce99_8bc8_4ece_91df_7dcb1d931c5e.slice/crio-b7306908e59c5178661b4b1873fa1fe085fe340d7d1d996dbefcf871003069b6 WatchSource:0}: Error finding container b7306908e59c5178661b4b1873fa1fe085fe340d7d1d996dbefcf871003069b6: Status 404 returned error can't find the container with id b7306908e59c5178661b4b1873fa1fe085fe340d7d1d996dbefcf871003069b6 Nov 03 16:44:36.284504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:36.276999 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:36.472739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:36.472662 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddf65ec-0b59-472f-abbd-21166349be99" path="/var/lib/kubelet/pods/0ddf65ec-0b59-472f-abbd-21166349be99/volumes" Nov 03 16:44:36.880810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:36.880771 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" event={"ID":"3292ce99-8bc8-4ece-91df-7dcb1d931c5e","Type":"ContainerStarted","Data":"b7306908e59c5178661b4b1873fa1fe085fe340d7d1d996dbefcf871003069b6"} Nov 03 16:44:38.053000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.052883 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:38.903836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.903799 4539 generic.go:356] "Generic (PLEG): container finished" podID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerID="c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f" exitCode=0 Nov 03 16:44:38.904048 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.903901 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" event={"ID":"a7c42f87-5b3f-4724-81b5-18be65528ec0","Type":"ContainerDied","Data":"c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f"} Nov 03 16:44:38.904392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.904371 4539 scope.go:122] "RemoveContainer" containerID="c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f" Nov 03 16:44:38.906101 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.906038 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:38.906220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.906129 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:38.906220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:38.906178 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:39.612631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:39.611776 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:39.612631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:39.611856 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:44:39.863257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:39.863158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:39.960413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:39.960076 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" event={"ID":"3292ce99-8bc8-4ece-91df-7dcb1d931c5e","Type":"ContainerStarted","Data":"1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a"} Nov 03 16:44:39.961723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:39.961456 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:39.964765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:39.964730 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" event={"ID":"a7c42f87-5b3f-4724-81b5-18be65528ec0","Type":"ContainerStarted","Data":"212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70"} Nov 03 16:44:40.037285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:40.037217 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" podStartSLOduration=1.629871685 podStartE2EDuration="5.037194814s" totalImagesPullingTime="3.407323129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:40.005943801 +0000 UTC m=+1823.699214823" watchObservedRunningTime="2026-11-03 16:44:40.037194814 +0000 UTC m=+1823.730465838" Nov 03 16:44:40.456152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:40.456118 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:44:40.457534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:40.457509 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:44:40.457704 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:40.457686 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 16:44:40.976473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:40.976046 4539 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k6pb9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:44:40.976473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:40.976092 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" podUID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:44:41.456673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:41.456638 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:44:41.456882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:41.456711 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 16:44:41.465828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:41.465799 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:41.642451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:41.633878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:41.982709 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:41.982683 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:44:42.292168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:42.292139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:42.456076 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:42.456034 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:44:42.456272 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:42.456142 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 16:44:42.686576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:42.686488 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:43.444503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:43.429823 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:43.460455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:43.460060 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:44:43.460455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:43.460124 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 16:44:43.725510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:43.724821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:44.351450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:44.351363 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:44.466447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:44.457867 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:44:44.466447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:44.457934 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 16:44:44.979640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:44.979595 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:45.456109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:45.456074 4539 patch_prober.go:29] interesting pod/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:44:45.456304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:45.456130 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 16:44:45.894201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:45.894163 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:46.469158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:46.469125 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:44:46.469689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:46.469669 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:44:46.470530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:46.470508 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:44:47.913356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:47.913318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:48.145493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:48.131020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:49.376741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:49.376711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:49.821785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:49.821746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:50.076736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:50.076632 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:50.356353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:50.356261 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:52.976488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:52.976454 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:53.267811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:53.267709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:53.791932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:53.791890 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:54.048248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:54.048114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:54.446399 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:54.446302 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:54.653473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:54.637043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:55.285460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:55.277014 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:55.898739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:55.898706 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:56.193172 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:44:56.193087 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:44:57.754449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:57.753656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:58.041455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:58.033025 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:58.273209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:58.273174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:44:59.478732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:44:59.478696 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:00.028204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.028131 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:00.427634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.418467 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:00.442265 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.428619 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b"] Nov 03 16:45:00.491772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.485005 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8"] Nov 03 16:45:00.493699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.492499 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b"] Nov 03 16:45:00.511481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.493988 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.511481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.497074 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.534549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.524405 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8"] Nov 03 16:45:00.534549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.524572 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6"] Nov 03 16:45:00.534549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.530294 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt"] Nov 03 16:45:00.535605 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.535170 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.539169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.538862 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6"] Nov 03 16:45:00.539169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.539009 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.585509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.585307 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:45:00.619687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.619644 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.619879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.619809 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.619948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.619897 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwch\" (UniqueName: \"kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.619995 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.619958 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjqc\" (UniqueName: \"kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc\") pod \"maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.620069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.620048 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2crc\" (UniqueName: \"kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.620116 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.620091 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.620194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.620177 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.620250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.620232 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssp5r\" (UniqueName: \"kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.649559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.647148 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt"] Nov 03 16:45:00.723875 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.723832 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.724081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.723932 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwch\" (UniqueName: \"kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.724081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.723992 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjqc\" (UniqueName: \"kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc\") pod \"maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.724201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.724088 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2crc\" (UniqueName: \"kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.724201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.724124 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.724201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.724195 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.724365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.724249 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssp5r\" (UniqueName: \"kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.724365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.724342 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.725348 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.725319 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.726154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.726132 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.745129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.745092 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.745979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.745954 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.769343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.769311 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjqc\" (UniqueName: \"kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc\") pod \"maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:00.770673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.770632 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwch\" (UniqueName: \"kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.774713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.774677 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2crc\" (UniqueName: \"kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.785597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.785559 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssp5r\" (UniqueName: \"kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:00.840582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.840537 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:00.945592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.927206 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:00.963709 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.963345 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:01.004449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:00.998684 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:01.177925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.177779 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b"] Nov 03 16:45:01.321996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.321962 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8"] Nov 03 16:45:01.323297 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:45:01.323271 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea5d4f9_0a5d_4949_8fa7_ddd4302e6ee8.slice/crio-2c7b2a6389622a2a70eb901c8ae078d716a65ab5422148ef9dffdfe6a4ce5229 WatchSource:0}: Error finding container 2c7b2a6389622a2a70eb901c8ae078d716a65ab5422148ef9dffdfe6a4ce5229: Status 404 returned error can't find the container with id 2c7b2a6389622a2a70eb901c8ae078d716a65ab5422148ef9dffdfe6a4ce5229 Nov 03 16:45:01.363405 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.363344 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6"] Nov 03 16:45:01.435507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.434323 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt"] Nov 03 16:45:01.637233 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.637190 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" event={"ID":"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8","Type":"ContainerStarted","Data":"2c7b2a6389622a2a70eb901c8ae078d716a65ab5422148ef9dffdfe6a4ce5229"} Nov 03 16:45:01.638812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.638784 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerStarted","Data":"6ad1eca291af7ab397dc88cca1881d07e2b51cf1c1b2cf71cb7252663d9fb2cc"} Nov 03 16:45:01.640499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.640471 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" event={"ID":"ee93ceb4-1fff-42be-9345-864b01adb451","Type":"ContainerStarted","Data":"6807ba0bec5c68ee11ebc3d53c59c071695fdc8ce0288ce1b38e0b57fb040ce4"} Nov 03 16:45:01.642024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:01.642000 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerStarted","Data":"e93df5c2f8967b4865ec2a39b6bf71d4259f3e2878dadfbfab3be7882c33f138"} Nov 03 16:45:02.388762 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.388529 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:02.655092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.655057 4539 generic.go:356] "Generic (PLEG): container finished" podID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerID="034fb9ba87c719dc3c0289750535b5eee23fd98db540d465825d6afa4c185750" exitCode=60 Nov 03 16:45:02.655594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.655192 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerDied","Data":"034fb9ba87c719dc3c0289750535b5eee23fd98db540d465825d6afa4c185750"} Nov 03 16:45:02.655594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.655557 4539 scope.go:122] "RemoveContainer" containerID="034fb9ba87c719dc3c0289750535b5eee23fd98db540d465825d6afa4c185750" Nov 03 16:45:02.657566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.657364 4539 generic.go:356] "Generic (PLEG): container finished" podID="ee93ceb4-1fff-42be-9345-864b01adb451" containerID="050935b7256b27506b4c23255d5717979ebfd5f3ea1fe717dad864093f093f6d" exitCode=60 Nov 03 16:45:02.657566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.657510 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" event={"ID":"ee93ceb4-1fff-42be-9345-864b01adb451","Type":"ContainerDied","Data":"050935b7256b27506b4c23255d5717979ebfd5f3ea1fe717dad864093f093f6d"} Nov 03 16:45:02.657789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.657751 4539 scope.go:122] "RemoveContainer" containerID="050935b7256b27506b4c23255d5717979ebfd5f3ea1fe717dad864093f093f6d" Nov 03 16:45:02.662418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.662113 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerStarted","Data":"4a353947913d5fcc8ea76d059d0ddfa2f79b7e8f86400cbb3f06d140b5235692"} Nov 03 16:45:02.666216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.666075 4539 generic.go:356] "Generic (PLEG): container finished" podID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerID="a2fc447a42c5deee0375d985383e04dc295149c152e90dd9897632c0575c5803" exitCode=60 Nov 03 16:45:02.666216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.666120 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" event={"ID":"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8","Type":"ContainerDied","Data":"a2fc447a42c5deee0375d985383e04dc295149c152e90dd9897632c0575c5803"} Nov 03 16:45:02.667393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.666475 4539 scope.go:122] "RemoveContainer" containerID="a2fc447a42c5deee0375d985383e04dc295149c152e90dd9897632c0575c5803" Nov 03 16:45:02.835447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:02.829188 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" podStartSLOduration=2.829168621 podStartE2EDuration="2.829168621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.827822502 +0000 UTC m=+1846.521093520" watchObservedRunningTime="2026-11-03 16:45:02.829168621 +0000 UTC m=+1846.522439645" Nov 03 16:45:03.049308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.049253 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd"] Nov 03 16:45:03.053712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.053684 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48007a6-4n4cd"] Nov 03 16:45:03.675595 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.675498 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" event={"ID":"ee93ceb4-1fff-42be-9345-864b01adb451","Type":"ContainerStarted","Data":"ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6"} Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.677205 4539 generic.go:356] "Generic (PLEG): container finished" podID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerID="4a353947913d5fcc8ea76d059d0ddfa2f79b7e8f86400cbb3f06d140b5235692" exitCode=60 Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.677283 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerDied","Data":"4a353947913d5fcc8ea76d059d0ddfa2f79b7e8f86400cbb3f06d140b5235692"} Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.677549 4539 scope.go:122] "RemoveContainer" containerID="4a353947913d5fcc8ea76d059d0ddfa2f79b7e8f86400cbb3f06d140b5235692" Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.679454 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" event={"ID":"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8","Type":"ContainerStarted","Data":"9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b"} Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.681280 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerStarted","Data":"d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45"} Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.720960 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" podStartSLOduration=3.720941124 podStartE2EDuration="3.720941124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.713990367 +0000 UTC m=+1847.407261389" watchObservedRunningTime="2026-11-03 16:45:03.720941124 +0000 UTC m=+1847.414212148" Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.757520 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" podStartSLOduration=3.7575064989999998 podStartE2EDuration="3.757506499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.752932979 +0000 UTC m=+1847.446204001" watchObservedRunningTime="2026-11-03 16:45:03.757506499 +0000 UTC m=+1847.450777521" Nov 03 16:45:03.952162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:03.798803 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" podStartSLOduration=3.798788181 podStartE2EDuration="3.798788181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:03.796027988 +0000 UTC m=+1847.489299009" watchObservedRunningTime="2026-11-03 16:45:03.798788181 +0000 UTC m=+1847.492059203" Nov 03 16:45:04.267955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.267858 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:04.470296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.470263 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e" path="/var/lib/kubelet/pods/c6b35f35-b4fa-43bf-bd0e-c617fef0ec7e/volumes" Nov 03 16:45:04.689777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.689743 4539 generic.go:356] "Generic (PLEG): container finished" podID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerID="9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b" exitCode=60 Nov 03 16:45:04.690250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.689820 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" event={"ID":"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8","Type":"ContainerDied","Data":"9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b"} Nov 03 16:45:04.690250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.689904 4539 scope.go:122] "RemoveContainer" containerID="a2fc447a42c5deee0375d985383e04dc295149c152e90dd9897632c0575c5803" Nov 03 16:45:04.690250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.690217 4539 scope.go:122] "RemoveContainer" containerID="9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b" Nov 03 16:45:04.690662 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:04.690634 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8_odh-ai-gateway-infra(4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" Nov 03 16:45:04.691869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.691846 4539 generic.go:356] "Generic (PLEG): container finished" podID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerID="d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45" exitCode=60 Nov 03 16:45:04.691965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.691928 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerDied","Data":"d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45"} Nov 03 16:45:04.692194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.692174 4539 scope.go:122] "RemoveContainer" containerID="d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45" Nov 03 16:45:04.692623 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:04.692602 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b_odh-ai-gateway-infra(bbf2b81f-db47-4153-bee2-9c148ff40f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" Nov 03 16:45:04.693518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.693495 4539 generic.go:356] "Generic (PLEG): container finished" podID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerID="7d3e69ab48de4e7f90a6d5625e5a6a86eec737e5b726ec72db3c7401ec26f98b" exitCode=0 Nov 03 16:45:04.693598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.693577 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" event={"ID":"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5","Type":"ContainerDied","Data":"7d3e69ab48de4e7f90a6d5625e5a6a86eec737e5b726ec72db3c7401ec26f98b"} Nov 03 16:45:04.693982 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.693963 4539 scope.go:122] "RemoveContainer" containerID="7d3e69ab48de4e7f90a6d5625e5a6a86eec737e5b726ec72db3c7401ec26f98b" Nov 03 16:45:04.695581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.695562 4539 generic.go:356] "Generic (PLEG): container finished" podID="ee93ceb4-1fff-42be-9345-864b01adb451" containerID="ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6" exitCode=60 Nov 03 16:45:04.695688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.695641 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" event={"ID":"ee93ceb4-1fff-42be-9345-864b01adb451","Type":"ContainerDied","Data":"ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6"} Nov 03 16:45:04.695861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.695843 4539 scope.go:122] "RemoveContainer" containerID="ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6" Nov 03 16:45:04.695948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.695911 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:45:04.696051 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.696009 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:45:04.696118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.696060 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:45:04.696170 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:04.696124 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6_odh-ai-gateway-infra(ee93ceb4-1fff-42be-9345-864b01adb451)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" Nov 03 16:45:04.699522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.699499 4539 generic.go:356] "Generic (PLEG): container finished" podID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerID="3363e8aa3004823936c12d1fe31b1832d91aa87bf130b80a45651ebb47d58bf3" exitCode=60 Nov 03 16:45:04.699761 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.699692 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerDied","Data":"3363e8aa3004823936c12d1fe31b1832d91aa87bf130b80a45651ebb47d58bf3"} Nov 03 16:45:04.699985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.699811 4539 scope.go:122] "RemoveContainer" containerID="3363e8aa3004823936c12d1fe31b1832d91aa87bf130b80a45651ebb47d58bf3" Nov 03 16:45:04.700158 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:04.700138 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt_odh-ai-gateway-infra(4c2a9fec-519d-4761-9056-e4da2c2528ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" Nov 03 16:45:04.701837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.701817 4539 generic.go:356] "Generic (PLEG): container finished" podID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerID="f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248" exitCode=0 Nov 03 16:45:04.701946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.701846 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" event={"ID":"131eb6da-b0bf-49df-9a3f-33480f237df0","Type":"ContainerDied","Data":"f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248"} Nov 03 16:45:04.702154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.702140 4539 scope.go:122] "RemoveContainer" containerID="f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248" Nov 03 16:45:04.706625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.706600 4539 scope.go:122] "RemoveContainer" containerID="034fb9ba87c719dc3c0289750535b5eee23fd98db540d465825d6afa4c185750" Nov 03 16:45:04.708064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.707602 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:45:04.708064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.707687 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:45:04.708064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.707734 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:45:04.729509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.729479 4539 scope.go:122] "RemoveContainer" containerID="050935b7256b27506b4c23255d5717979ebfd5f3ea1fe717dad864093f093f6d" Nov 03 16:45:04.758912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:04.758893 4539 scope.go:122] "RemoveContainer" containerID="4a353947913d5fcc8ea76d059d0ddfa2f79b7e8f86400cbb3f06d140b5235692" Nov 03 16:45:05.394514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.394483 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:45:05.770187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.770081 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" event={"ID":"131eb6da-b0bf-49df-9a3f-33480f237df0","Type":"ContainerStarted","Data":"79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36"} Nov 03 16:45:05.773282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.773257 4539 scope.go:122] "RemoveContainer" containerID="9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b" Nov 03 16:45:05.773646 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:05.773620 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8_odh-ai-gateway-infra(4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" Nov 03 16:45:05.776083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.776059 4539 scope.go:122] "RemoveContainer" containerID="d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45" Nov 03 16:45:05.776460 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:05.776400 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b_odh-ai-gateway-infra(bbf2b81f-db47-4153-bee2-9c148ff40f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" Nov 03 16:45:05.778626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.778597 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" event={"ID":"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5","Type":"ContainerStarted","Data":"967dd21b2da893055df769acb6b642fdf7b333e416f9d3741ea5213e52ecf8df"} Nov 03 16:45:05.780617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.780602 4539 scope.go:122] "RemoveContainer" containerID="ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6" Nov 03 16:45:05.780868 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:05.780842 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6_odh-ai-gateway-infra(ee93ceb4-1fff-42be-9345-864b01adb451)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" Nov 03 16:45:05.782174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.782156 4539 scope.go:122] "RemoveContainer" containerID="3363e8aa3004823936c12d1fe31b1832d91aa87bf130b80a45651ebb47d58bf3" Nov 03 16:45:05.782439 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:05.782404 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt_odh-ai-gateway-infra(4c2a9fec-519d-4761-9056-e4da2c2528ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" Nov 03 16:45:05.886370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:05.885663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:06.079996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.079959 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:06.083736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.083712 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:45:06.084761 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.084737 4539 patch_prober.go:29] interesting pod/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:45:06.084939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.084913 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 16:45:06.161457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.158198 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:45:06.182634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.182582 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 16:45:06.182986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.182918 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 16:45:06.401668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:06.401568 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:45:07.061906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:07.061855 4539 patch_prober.go:29] interesting pod/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:45:07.062468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:07.061924 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 16:45:07.160909 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:07.160878 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 16:45:07.161142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:07.161120 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 16:45:07.988861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:07.988832 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:08.060920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:08.060876 4539 patch_prober.go:29] interesting pod/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:45:08.061247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:08.061212 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 16:45:08.159372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:08.159330 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 16:45:08.160023 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:08.159979 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 16:45:08.992306 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:08.992258 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:09.215444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:09.203876 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 16:45:09.215444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:09.203936 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 16:45:09.215444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:09.204838 4539 patch_prober.go:29] interesting pod/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:45:09.215444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:09.204904 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 16:45:09.319156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:09.319125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:09.860450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:09.851479 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:10.056266 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:10.056231 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:10.063935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:10.063903 4539 patch_prober.go:29] interesting pod/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:45:10.064215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:10.064186 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 16:45:10.159738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:10.159642 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 16:45:10.159989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:10.159962 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 16:45:10.469494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:10.469357 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:11.042449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:11.041515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:11.064969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:11.064934 4539 patch_prober.go:29] interesting pod/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 03 16:45:11.065277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:11.065246 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 03 16:45:11.178482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:11.175943 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 16:45:11.178482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:11.176032 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 16:45:12.065156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.065123 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:45:12.066370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.066348 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:45:12.162588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.162538 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:45:12.163338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.163315 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:45:12.163990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.163931 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:45:12.317005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.316918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:12.468723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.468676 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:12.618600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.618341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:12.790640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:12.790612 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:13.501171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:13.501137 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:13.958365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:13.958331 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:14.450340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:14.450287 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:16.329921 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:16.329830 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:16.466180 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:16.466139 4539 scope.go:122] "RemoveContainer" containerID="d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45" Nov 03 16:45:17.015396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:17.015298 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerStarted","Data":"9d0dac4ef35f39a8c49e2405931d438eb17c717d4585862a3bbd423c1e9834f7"} Nov 03 16:45:17.391633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:17.391598 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:17.461893 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:17.461864 4539 scope.go:122] "RemoveContainer" containerID="9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b" Nov 03 16:45:17.462097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:17.462023 4539 scope.go:122] "RemoveContainer" containerID="3363e8aa3004823936c12d1fe31b1832d91aa87bf130b80a45651ebb47d58bf3" Nov 03 16:45:17.462167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:17.462125 4539 scope.go:122] "RemoveContainer" containerID="ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6" Nov 03 16:45:17.584906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:17.584836 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:18.026435 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.026388 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerStarted","Data":"3db0879079441333e45f0020f45576410c66b3e546d6d0c2ad9647351461f481"} Nov 03 16:45:18.028448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.028406 4539 generic.go:356] "Generic (PLEG): container finished" podID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerID="9d0dac4ef35f39a8c49e2405931d438eb17c717d4585862a3bbd423c1e9834f7" exitCode=60 Nov 03 16:45:18.028566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.028471 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerDied","Data":"9d0dac4ef35f39a8c49e2405931d438eb17c717d4585862a3bbd423c1e9834f7"} Nov 03 16:45:18.028566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.028537 4539 scope.go:122] "RemoveContainer" containerID="d80c8423f4a584d8ae911100538323e9cb95df9043176dac328c9b3fa9d48d45" Nov 03 16:45:18.028960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.028939 4539 scope.go:122] "RemoveContainer" containerID="9d0dac4ef35f39a8c49e2405931d438eb17c717d4585862a3bbd423c1e9834f7" Nov 03 16:45:18.029323 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:18.029303 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b_odh-ai-gateway-infra(bbf2b81f-db47-4153-bee2-9c148ff40f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" Nov 03 16:45:18.092071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.091989 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b"] Nov 03 16:45:18.612478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.612445 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:18.781873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.781841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:18.945168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:18.945133 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:19.044679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.044576 4539 generic.go:356] "Generic (PLEG): container finished" podID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerID="3db0879079441333e45f0020f45576410c66b3e546d6d0c2ad9647351461f481" exitCode=60 Nov 03 16:45:19.044679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.044645 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerDied","Data":"3db0879079441333e45f0020f45576410c66b3e546d6d0c2ad9647351461f481"} Nov 03 16:45:19.044902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.044734 4539 scope.go:122] "RemoveContainer" containerID="3363e8aa3004823936c12d1fe31b1832d91aa87bf130b80a45651ebb47d58bf3" Nov 03 16:45:19.048725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.048701 4539 scope.go:122] "RemoveContainer" containerID="3db0879079441333e45f0020f45576410c66b3e546d6d0c2ad9647351461f481" Nov 03 16:45:19.048863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.048838 4539 generic.go:356] "Generic (PLEG): container finished" podID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerID="5066fd7a9a6190bcfa13a0e27d8c82f8cdc4b4840bc46c23de836f5ff9c4099e" exitCode=60 Nov 03 16:45:19.048963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.048927 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" event={"ID":"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8","Type":"ContainerDied","Data":"5066fd7a9a6190bcfa13a0e27d8c82f8cdc4b4840bc46c23de836f5ff9c4099e"} Nov 03 16:45:19.049512 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:19.049111 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt_odh-ai-gateway-infra(4c2a9fec-519d-4761-9056-e4da2c2528ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" Nov 03 16:45:19.049512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.049471 4539 scope.go:122] "RemoveContainer" containerID="5066fd7a9a6190bcfa13a0e27d8c82f8cdc4b4840bc46c23de836f5ff9c4099e" Nov 03 16:45:19.049846 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:19.049821 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8_odh-ai-gateway-infra(4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" Nov 03 16:45:19.054982 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.054943 4539 generic.go:356] "Generic (PLEG): container finished" podID="ee93ceb4-1fff-42be-9345-864b01adb451" containerID="f14c7fd82f868a21e85263d8d2e46dcc555fd89c066a49bf2dffdb4e6fd225de" exitCode=60 Nov 03 16:45:19.055132 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.055067 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" event={"ID":"ee93ceb4-1fff-42be-9345-864b01adb451","Type":"ContainerDied","Data":"f14c7fd82f868a21e85263d8d2e46dcc555fd89c066a49bf2dffdb4e6fd225de"} Nov 03 16:45:19.055606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.055583 4539 scope.go:122] "RemoveContainer" containerID="f14c7fd82f868a21e85263d8d2e46dcc555fd89c066a49bf2dffdb4e6fd225de" Nov 03 16:45:19.056010 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:19.055988 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6_odh-ai-gateway-infra(ee93ceb4-1fff-42be-9345-864b01adb451)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" Nov 03 16:45:19.062481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.062456 4539 scope.go:122] "RemoveContainer" containerID="9a4baa8f46792887df04b7eab8f7dc859faa672b9f6865c3c5ceb8e28908708b" Nov 03 16:45:19.084640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.084610 4539 scope.go:122] "RemoveContainer" containerID="ca98d34c7cef4b8fdc214bae8f2203dd87ab14f926b27e119a30024138716db6" Nov 03 16:45:19.134870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.134840 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt"] Nov 03 16:45:19.141301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.141277 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:19.269854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.269829 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:19.305085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.304992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:19.458556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.445509 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca\") pod \"bbf2b81f-db47-4153-bee2-9c148ff40f59\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " Nov 03 16:45:19.566376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.459178 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca" pod "bbf2b81f-db47-4153-bee2-9c148ff40f59" (UID: "bbf2b81f-db47-4153-bee2-9c148ff40f59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.571029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.569604 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch\" (UniqueName: \"kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch\") pod \"bbf2b81f-db47-4153-bee2-9c148ff40f59\" (UID: \"bbf2b81f-db47-4153-bee2-9c148ff40f59\") " Nov 03 16:45:19.583242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.583200 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch" pod "bbf2b81f-db47-4153-bee2-9c148ff40f59" (UID: "bbf2b81f-db47-4153-bee2-9c148ff40f59"). InnerVolumeSpecName "kube-api-access-vcwch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.598094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.598063 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf2b81f-db47-4153-bee2-9c148ff40f59-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.598094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:19.598093 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwch\" (UniqueName: \"kubernetes.io/projected/bbf2b81f-db47-4153-bee2-9c148ff40f59-kube-api-access-vcwch\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:20.065834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.065800 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" Nov 03 16:45:20.065834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.065820 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b" event={"ID":"bbf2b81f-db47-4153-bee2-9c148ff40f59","Type":"ContainerDied","Data":"6ad1eca291af7ab397dc88cca1881d07e2b51cf1c1b2cf71cb7252663d9fb2cc"} Nov 03 16:45:20.066397 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.065877 4539 scope.go:122] "RemoveContainer" containerID="9d0dac4ef35f39a8c49e2405931d438eb17c717d4585862a3bbd423c1e9834f7" Nov 03 16:45:20.115203 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.115175 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b"] Nov 03 16:45:20.126717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.126686 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895405-krr8b"] Nov 03 16:45:20.166652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.166609 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8"] Nov 03 16:45:20.239364 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.239332 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6"] Nov 03 16:45:20.287695 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.287115 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:20.316211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.316136 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r\" (UniqueName: \"kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r\") pod \"4c2a9fec-519d-4761-9056-e4da2c2528ae\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " Nov 03 16:45:20.316410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.316390 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca\") pod \"4c2a9fec-519d-4761-9056-e4da2c2528ae\" (UID: \"4c2a9fec-519d-4761-9056-e4da2c2528ae\") " Nov 03 16:45:20.318480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.318446 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca" pod "4c2a9fec-519d-4761-9056-e4da2c2528ae" (UID: "4c2a9fec-519d-4761-9056-e4da2c2528ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:20.322108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.321105 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r" pod "4c2a9fec-519d-4761-9056-e4da2c2528ae" (UID: "4c2a9fec-519d-4761-9056-e4da2c2528ae"). InnerVolumeSpecName "kube-api-access-ssp5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:20.418612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.418580 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssp5r\" (UniqueName: \"kubernetes.io/projected/4c2a9fec-519d-4761-9056-e4da2c2528ae-kube-api-access-ssp5r\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:20.418809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.418793 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2a9fec-519d-4761-9056-e4da2c2528ae-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:20.423551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.423525 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:20.451937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.451913 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:20.472094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.472057 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" path="/var/lib/kubelet/pods/bbf2b81f-db47-4153-bee2-9c148ff40f59/volumes" Nov 03 16:45:20.520155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.520093 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc\" (UniqueName: \"kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc\") pod \"ee93ceb4-1fff-42be-9345-864b01adb451\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " Nov 03 16:45:20.520361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.520298 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc\" (UniqueName: \"kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc\") pod \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " Nov 03 16:45:20.520447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.520409 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca\") pod \"ee93ceb4-1fff-42be-9345-864b01adb451\" (UID: \"ee93ceb4-1fff-42be-9345-864b01adb451\") " Nov 03 16:45:20.520514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.520472 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca\") pod \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\" (UID: \"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8\") " Nov 03 16:45:20.522278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.520842 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca" pod "ee93ceb4-1fff-42be-9345-864b01adb451" (UID: "ee93ceb4-1fff-42be-9345-864b01adb451"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:20.522278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.520963 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca" pod "4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" (UID: "4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:20.522278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.522197 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee93ceb4-1fff-42be-9345-864b01adb451-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:20.522278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.522220 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:20.523351 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.523324 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc" pod "4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" (UID: "4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8"). InnerVolumeSpecName "kube-api-access-k2crc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:20.524665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.524642 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc" pod "ee93ceb4-1fff-42be-9345-864b01adb451" (UID: "ee93ceb4-1fff-42be-9345-864b01adb451"). InnerVolumeSpecName "kube-api-access-qbjqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:20.623323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.623282 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2crc\" (UniqueName: \"kubernetes.io/projected/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8-kube-api-access-k2crc\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:20.623323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:20.623315 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjqc\" (UniqueName: \"kubernetes.io/projected/ee93ceb4-1fff-42be-9345-864b01adb451-kube-api-access-qbjqc\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:21.057096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.057043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:21.084906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.084840 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" event={"ID":"ee93ceb4-1fff-42be-9345-864b01adb451","Type":"ContainerDied","Data":"6807ba0bec5c68ee11ebc3d53c59c071695fdc8ce0288ce1b38e0b57fb040ce4"} Nov 03 16:45:21.087478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.084922 4539 scope.go:122] "RemoveContainer" containerID="f14c7fd82f868a21e85263d8d2e46dcc555fd89c066a49bf2dffdb4e6fd225de" Nov 03 16:45:21.087478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.085036 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6" Nov 03 16:45:21.099320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.099268 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" event={"ID":"4c2a9fec-519d-4761-9056-e4da2c2528ae","Type":"ContainerDied","Data":"e93df5c2f8967b4865ec2a39b6bf71d4259f3e2878dadfbfab3be7882c33f138"} Nov 03 16:45:21.099677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.099655 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt" Nov 03 16:45:21.137355 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.137325 4539 scope.go:122] "RemoveContainer" containerID="3db0879079441333e45f0020f45576410c66b3e546d6d0c2ad9647351461f481" Nov 03 16:45:21.152471 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.152440 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" Nov 03 16:45:21.158172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.152681 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8" event={"ID":"4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8","Type":"ContainerDied","Data":"2c7b2a6389622a2a70eb901c8ae078d716a65ab5422148ef9dffdfe6a4ce5229"} Nov 03 16:45:21.159535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.159512 4539 scope.go:122] "RemoveContainer" containerID="5066fd7a9a6190bcfa13a0e27d8c82f8cdc4b4840bc46c23de836f5ff9c4099e" Nov 03 16:45:21.197130 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.197090 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6"] Nov 03 16:45:21.203881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.203840 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-156c2a-29895405-mklw6"] Nov 03 16:45:21.229701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.229668 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8"] Nov 03 16:45:21.234229 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.234201 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895405-dqlc8"] Nov 03 16:45:21.258260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.258227 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt"] Nov 03 16:45:21.266482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:21.266456 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895405-w8svt"] Nov 03 16:45:22.472006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:22.471963 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" path="/var/lib/kubelet/pods/4c2a9fec-519d-4761-9056-e4da2c2528ae/volumes" Nov 03 16:45:22.472609 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:22.472591 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" path="/var/lib/kubelet/pods/4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8/volumes" Nov 03 16:45:22.473170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:22.473152 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" path="/var/lib/kubelet/pods/ee93ceb4-1fff-42be-9345-864b01adb451/volumes" Nov 03 16:45:22.627734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:22.627686 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:22.782223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:22.782192 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:24.472769 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:24.472727 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:25.988409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:25.988372 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:26.169452 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:26.168946 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:45:26.498658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:26.498622 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh"] Nov 03 16:45:26.513963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:26.513901 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" podUID="8f531f1d-6b44-4499-b1c2-cac8a28ef78d" containerName="maas-api" containerID="cri-o://27bb74f1fd9f8560f364cdea1c1e32f2f66afee531c49d535437002d928ca389" gracePeriod=30 Nov 03 16:45:26.730546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:26.716273 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:27.415967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.391811 4539 generic.go:356] "Generic (PLEG): container finished" podID="8f531f1d-6b44-4499-b1c2-cac8a28ef78d" containerID="27bb74f1fd9f8560f364cdea1c1e32f2f66afee531c49d535437002d928ca389" exitCode=0 Nov 03 16:45:27.415967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.392087 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" event={"ID":"8f531f1d-6b44-4499-b1c2-cac8a28ef78d","Type":"ContainerDied","Data":"27bb74f1fd9f8560f364cdea1c1e32f2f66afee531c49d535437002d928ca389"} Nov 03 16:45:27.532009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.531978 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:45:27.600992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.600906 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca\") pod \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " Nov 03 16:45:27.601168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.600996 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs\") pod \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " Nov 03 16:45:27.601168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.601089 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc\" (UniqueName: \"kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc\") pod \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " Nov 03 16:45:27.601283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.601210 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\") pod \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\" (UID: \"8f531f1d-6b44-4499-b1c2-cac8a28ef78d\") " Nov 03 16:45:27.602293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.602177 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca" pod "8f531f1d-6b44-4499-b1c2-cac8a28ef78d" (UID: "8f531f1d-6b44-4499-b1c2-cac8a28ef78d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:27.604545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.604504 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs" pod "8f531f1d-6b44-4499-b1c2-cac8a28ef78d" (UID: "8f531f1d-6b44-4499-b1c2-cac8a28ef78d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:27.612614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.605936 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc" pod "8f531f1d-6b44-4499-b1c2-cac8a28ef78d" (UID: "8f531f1d-6b44-4499-b1c2-cac8a28ef78d"). InnerVolumeSpecName "kube-api-access-2nmtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:27.612614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.605975 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls" pod "8f531f1d-6b44-4499-b1c2-cac8a28ef78d" (UID: "8f531f1d-6b44-4499-b1c2-cac8a28ef78d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:27.703451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.703394 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.703451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.703447 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.703451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.703457 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmtc\" (UniqueName: \"kubernetes.io/projected/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-kube-api-access-2nmtc\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.703720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.703466 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f531f1d-6b44-4499-b1c2-cac8a28ef78d-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.931476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:27.927809 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:28.401147 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:28.401121 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" Nov 03 16:45:28.401343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:28.401118 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh" event={"ID":"8f531f1d-6b44-4499-b1c2-cac8a28ef78d","Type":"ContainerDied","Data":"970b89c7fc9ecc06b2c418afdc81d671604031b5786088eaa483c0b1d008e580"} Nov 03 16:45:28.401343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:28.401263 4539 scope.go:122] "RemoveContainer" containerID="27bb74f1fd9f8560f364cdea1c1e32f2f66afee531c49d535437002d928ca389" Nov 03 16:45:28.469491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:28.469464 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh"] Nov 03 16:45:28.473999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:28.473970 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bffe15-6f58f8996d-z79wh"] Nov 03 16:45:29.684297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:29.684249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:30.102058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:30.102021 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm"] Nov 03 16:45:30.128376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:30.128339 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb11b61a-qqcwm"] Nov 03 16:45:30.469308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:30.469232 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1e60e1-92bb-4dd0-a43b-a3f368c2faec" path="/var/lib/kubelet/pods/2f1e60e1-92bb-4dd0-a43b-a3f368c2faec/volumes" Nov 03 16:45:30.469683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:30.469668 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f531f1d-6b44-4499-b1c2-cac8a28ef78d" path="/var/lib/kubelet/pods/8f531f1d-6b44-4499-b1c2-cac8a28ef78d/volumes" Nov 03 16:45:31.012689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:31.012609 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:31.311607 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:31.311575 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:32.095856 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:32.095812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:32.264652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:32.264618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:32.584452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:32.584010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:33.432356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:33.432310 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:34.630783 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:34.630749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:36.722802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:36.722756 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:36.992178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:36.992088 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:37.571874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:37.571840 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:37.929765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:37.929677 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz"] Nov 03 16:45:37.930588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:37.930557 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" containerID="cri-o://75e8f42b37db4b9c8583fa73ee324d29b1e2e67590d34f92e6b2886d1d59a7a5" gracePeriod=30 Nov 03 16:45:37.943101 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:37.943066 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr"] Nov 03 16:45:37.943566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:37.943537 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" containerID="cri-o://63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027" gracePeriod=30 Nov 03 16:45:38.032485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:38.032445 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:38.935036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:38.935003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:39.484389 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:39.484342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:39.659952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:39.659917 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:40.216621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:40.214216 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:40.389892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:40.389862 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:41.508499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:41.508463 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:41.732618 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:41.732581 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:41.885315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:41.885276 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:42.053478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:42.053446 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:43.209097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.209062 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:43.258702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.258672 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:45:43.327006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.326970 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.349536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.331919 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.370561 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.370648 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.370750 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.370851 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.370902 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.370930 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.371043 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo\") pod \"42f4ba74-0cc1-4ab0-a165-c522537165da\" (UID: \"42f4ba74-0cc1-4ab0-a165-c522537165da\") " Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.333879 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.379473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.371745 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.409452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.387563 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.440985 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.461512 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.462304 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.462331 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f4ba74-0cc1-4ab0-a165-c522537165da-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.462347 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.462361 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.473256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.462374 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.499508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.485629 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "kube-api-access-6hrhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:43.499508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.490544 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.513871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.513835 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:43.525950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.525093 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:45:43.559920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.540597 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token" pod "42f4ba74-0cc1-4ab0-a165-c522537165da" (UID: "42f4ba74-0cc1-4ab0-a165-c522537165da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:43.591462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.590956 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.591462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.591000 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrhm\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-kube-api-access-6hrhm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.591462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.591015 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.591462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.591028 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f4ba74-0cc1-4ab0-a165-c522537165da-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.679257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.679170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:43.687758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.687721 4539 generic.go:356] "Generic (PLEG): container finished" podID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerID="75e8f42b37db4b9c8583fa73ee324d29b1e2e67590d34f92e6b2886d1d59a7a5" exitCode=0 Nov 03 16:45:43.688133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.688104 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" event={"ID":"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326","Type":"ContainerDied","Data":"75e8f42b37db4b9c8583fa73ee324d29b1e2e67590d34f92e6b2886d1d59a7a5"} Nov 03 16:45:43.688361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.688346 4539 scope.go:122] "RemoveContainer" containerID="a8b0be3638527d0192e26b5c76a250e77079b4885ade2f343b6645e8a76fb798" Nov 03 16:45:43.694289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.694259 4539 generic.go:356] "Generic (PLEG): container finished" podID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerID="63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027" exitCode=0 Nov 03 16:45:43.694874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.694623 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" Nov 03 16:45:43.695467 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.694643 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" event={"ID":"42f4ba74-0cc1-4ab0-a165-c522537165da","Type":"ContainerDied","Data":"63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027"} Nov 03 16:45:43.695639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.695620 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr" event={"ID":"42f4ba74-0cc1-4ab0-a165-c522537165da","Type":"ContainerDied","Data":"465e20675a5047a47b92033e2ddf0444ba536f35cbb836d7b352d7d197c6249b"} Nov 03 16:45:43.803536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.787830 4539 scope.go:122] "RemoveContainer" containerID="63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027" Nov 03 16:45:43.849008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.848967 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr"] Nov 03 16:45:43.861456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.860486 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-qd4rr"] Nov 03 16:45:43.889019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.888624 4539 scope.go:122] "RemoveContainer" containerID="63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027" Nov 03 16:45:43.890079 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:43.889970 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027\": container with ID starting with 63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027 not found: ID does not exist" containerID="63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027" Nov 03 16:45:43.890079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:43.890016 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027"} err="failed to get container status \"63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027\": rpc error: code = NotFound desc = could not find container \"63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027\": container with ID starting with 63c2187ca55e74bf16e2787a6d00fe2eb088e669ef17b8531d031df8f360a027 not found: ID does not exist" Nov 03 16:45:44.132826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.132794 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:45:44.225071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225030 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225128 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225245 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225278 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225344 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225474 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225538 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.225627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225566 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.226043 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.225676 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket\") pod \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\" (UID: \"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326\") " Nov 03 16:45:44.226568 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.226542 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.226977 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.226950 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.238061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.231576 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.238061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.232813 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "kube-api-access-d5vrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:44.238061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.233318 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.238061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.234124 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:44.246036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.246002 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:44.263994 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.263957 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.265316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.265289 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo" pod "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" (UID: "9e27eaf5-a18a-4bef-b46b-aa4ac77ee326"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333026 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333065 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333078 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vrx\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-kube-api-access-d5vrx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333089 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333100 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333111 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333122 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333135 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.333226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.333145 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.473353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.473258 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" path="/var/lib/kubelet/pods/42f4ba74-0cc1-4ab0-a165-c522537165da/volumes" Nov 03 16:45:44.667741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.667680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:44.712481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.712404 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" Nov 03 16:45:44.713632 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.712390 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz" event={"ID":"9e27eaf5-a18a-4bef-b46b-aa4ac77ee326","Type":"ContainerDied","Data":"0418d05636e18447cff6c65919b75309bdca85cf06c65b0108ad0393c0500518"} Nov 03 16:45:44.713786 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.713698 4539 scope.go:122] "RemoveContainer" containerID="75e8f42b37db4b9c8583fa73ee324d29b1e2e67590d34f92e6b2886d1d59a7a5" Nov 03 16:45:44.809143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.809104 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz"] Nov 03 16:45:44.815958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.815910 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-p5fqz"] Nov 03 16:45:44.938570 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:44.938535 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:45.093493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:45.093461 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:46.206315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:46.206283 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:46.368770 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:46.368726 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:46.624737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:46.624700 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" path="/var/lib/kubelet/pods/9e27eaf5-a18a-4bef-b46b-aa4ac77ee326/volumes" Nov 03 16:45:47.781943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:47.781911 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:47.997361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:47.997329 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:48.152358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:48.152267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:48.743827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:48.743787 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:49.085527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:49.085497 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6"] Nov 03 16:45:49.085984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:49.085773 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" containerID="cri-o://79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36" gracePeriod=30 Nov 03 16:45:49.254454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:49.254382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:49.426704 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:49.426598 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:50.092657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:50.092623 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:50.632980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:50.632942 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:51.132034 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:51.132009 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:51.456614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:51.456521 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:51.922078 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:51.922034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:54.164130 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.164095 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:54.412079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.411681 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:45:54.578612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578579 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.578826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578637 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.578826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578676 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.578826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578711 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.578826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578774 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.578826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578823 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.579099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578928 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.579099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.578976 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:54.579099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.579050 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.579257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.579171 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket\") pod \"131eb6da-b0bf-49df-9a3f-33480f237df0\" (UID: \"131eb6da-b0bf-49df-9a3f-33480f237df0\") " Nov 03 16:45:54.579318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.579262 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.579616 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.579632 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.579751 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.580310 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.580333 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.580349 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.580363 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131eb6da-b0bf-49df-9a3f-33480f237df0-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.580402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.580376 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.581492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.581461 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:45:54.582321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.582298 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "kube-api-access-zqg8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:54.582391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.582332 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:54.582525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.582507 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy" pod "131eb6da-b0bf-49df-9a3f-33480f237df0" (UID: "131eb6da-b0bf-49df-9a3f-33480f237df0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:54.681723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.681642 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.681723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.681669 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.681723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.681678 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqg8n\" (UniqueName: \"kubernetes.io/projected/131eb6da-b0bf-49df-9a3f-33480f237df0-kube-api-access-zqg8n\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.681723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.681686 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131eb6da-b0bf-49df-9a3f-33480f237df0-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:45:54.832662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.832620 4539 generic.go:356] "Generic (PLEG): container finished" podID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerID="79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36" exitCode=0 Nov 03 16:45:54.834091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.834057 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" event={"ID":"131eb6da-b0bf-49df-9a3f-33480f237df0","Type":"ContainerDied","Data":"79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36"} Nov 03 16:45:54.834332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.834314 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" event={"ID":"131eb6da-b0bf-49df-9a3f-33480f237df0","Type":"ContainerDied","Data":"eeb0d8a4380a55e37bef59ae2646b8b875b6e68e4f12d78de29ee742006796da"} Nov 03 16:45:54.834479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.834465 4539 scope.go:122] "RemoveContainer" containerID="79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36" Nov 03 16:45:54.834757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.834741 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6" Nov 03 16:45:54.926214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.925854 4539 scope.go:122] "RemoveContainer" containerID="f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248" Nov 03 16:45:54.932630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.932562 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6"] Nov 03 16:45:54.941935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.941906 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-73cc79-openshift-default-7fcdf96759-6bhc6"] Nov 03 16:45:54.948078 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.948056 4539 scope.go:122] "RemoveContainer" containerID="79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36" Nov 03 16:45:54.948469 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:54.948438 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36\": container with ID starting with 79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36 not found: ID does not exist" containerID="79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36" Nov 03 16:45:54.948538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.948481 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36"} err="failed to get container status \"79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36\": rpc error: code = NotFound desc = could not find container \"79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36\": container with ID starting with 79ef6413a2b60c4efe74a277720e027df9470817fef3e23a7d90f90f7e4ade36 not found: ID does not exist" Nov 03 16:45:54.948538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.948510 4539 scope.go:122] "RemoveContainer" containerID="f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248" Nov 03 16:45:54.948896 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:54.948874 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248\": container with ID starting with f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248 not found: ID does not exist" containerID="f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248" Nov 03 16:45:54.948956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:54.948905 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248"} err="failed to get container status \"f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248\": rpc error: code = NotFound desc = could not find container \"f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248\": container with ID starting with f4b313590f93ca9964e3e7684e66a954c61c7354e2fe872f7685b180fc654248 not found: ID does not exist" Nov 03 16:45:55.359332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:55.359301 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:55.724775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:55.724673 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:55.889203 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:55.889165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:56.157999 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:45:56.157964 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:45:56.318004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:56.317968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:56.472883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:56.472804 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" path="/var/lib/kubelet/pods/131eb6da-b0bf-49df-9a3f-33480f237df0/volumes" Nov 03 16:45:57.096401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:57.096352 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:57.547338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:57.547304 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:57.699689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:57.699659 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:45:58.714827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:45:58.714794 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:00.612297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:00.612265 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:02.420461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:02.420410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:02.593951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:02.593909 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:02.998116 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:02.998081 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:03.292931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.292775 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:03.466263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.466225 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg"] Nov 03 16:46:03.466741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.466491 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" containerID="cri-o://d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac" gracePeriod=30 Nov 03 16:46:03.477117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.477094 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg"] Nov 03 16:46:03.477297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.477272 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" containerID="cri-o://c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636" gracePeriod=30 Nov 03 16:46:03.480958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.480938 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k"] Nov 03 16:46:03.481129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:03.481111 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" containerID="cri-o://93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d" gracePeriod=30 Nov 03 16:46:04.154861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:04.154829 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:04.301940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:04.301907 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:04.499380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:04.499288 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:04.639245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:04.639200 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:05.209250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.209214 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589"] Nov 03 16:46:05.209513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.209485 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" podUID="fb7d6160-b41b-400b-9316-35478bbe98ce" containerName="maas-api" containerID="cri-o://c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6" gracePeriod=30 Nov 03 16:46:05.288719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.288647 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:05.695281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.695255 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:46:05.811561 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.807395 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np\" (UniqueName: \"kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np\") pod \"fb7d6160-b41b-400b-9316-35478bbe98ce\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " Nov 03 16:46:05.812033 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.811995 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls\") pod \"fb7d6160-b41b-400b-9316-35478bbe98ce\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " Nov 03 16:46:05.812714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.812695 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs\") pod \"fb7d6160-b41b-400b-9316-35478bbe98ce\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " Nov 03 16:46:05.812912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.812886 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca\") pod \"fb7d6160-b41b-400b-9316-35478bbe98ce\" (UID: \"fb7d6160-b41b-400b-9316-35478bbe98ce\") " Nov 03 16:46:05.823176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.822924 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls" pod "fb7d6160-b41b-400b-9316-35478bbe98ce" (UID: "fb7d6160-b41b-400b-9316-35478bbe98ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:46:05.823861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.823484 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca" pod "fb7d6160-b41b-400b-9316-35478bbe98ce" (UID: "fb7d6160-b41b-400b-9316-35478bbe98ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:05.852447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.839746 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np" pod "fb7d6160-b41b-400b-9316-35478bbe98ce" (UID: "fb7d6160-b41b-400b-9316-35478bbe98ce"). InnerVolumeSpecName "kube-api-access-v49np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:05.852817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.852752 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs" pod "fb7d6160-b41b-400b-9316-35478bbe98ce" (UID: "fb7d6160-b41b-400b-9316-35478bbe98ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:46:05.915627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.915500 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:05.915627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.915528 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7d6160-b41b-400b-9316-35478bbe98ce-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:05.915627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.915538 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d6160-b41b-400b-9316-35478bbe98ce-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:05.915627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.915549 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49np\" (UniqueName: \"kubernetes.io/projected/fb7d6160-b41b-400b-9316-35478bbe98ce-kube-api-access-v49np\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:05.959904 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.959868 4539 generic.go:356] "Generic (PLEG): container finished" podID="fb7d6160-b41b-400b-9316-35478bbe98ce" containerID="c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6" exitCode=0 Nov 03 16:46:05.960086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.959916 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" Nov 03 16:46:05.960086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.959945 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" event={"ID":"fb7d6160-b41b-400b-9316-35478bbe98ce","Type":"ContainerDied","Data":"c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6"} Nov 03 16:46:05.960086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.960003 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589" event={"ID":"fb7d6160-b41b-400b-9316-35478bbe98ce","Type":"ContainerDied","Data":"befdb2bb4d5d935c26cc78e4aa796cceace4ba64dfc5c1638c8e51e40a8026b0"} Nov 03 16:46:05.960086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.960033 4539 scope.go:122] "RemoveContainer" containerID="c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6" Nov 03 16:46:05.972957 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.972937 4539 scope.go:122] "RemoveContainer" containerID="c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6" Nov 03 16:46:05.973258 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:46:05.973230 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6\": container with ID starting with c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6 not found: ID does not exist" containerID="c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6" Nov 03 16:46:05.973346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:05.973269 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6"} err="failed to get container status \"c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6\": rpc error: code = NotFound desc = could not find container \"c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6\": container with ID starting with c1e97467854508f00b1f007a3975c328da0799c45a0bdd276be5f474ad1580a6 not found: ID does not exist" Nov 03 16:46:06.010687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:06.010660 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589"] Nov 03 16:46:06.018242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:06.018223 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db3c6b-67bfd9cff5-2p589"] Nov 03 16:46:06.269149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:06.269115 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:06.471396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:06.471362 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7d6160-b41b-400b-9316-35478bbe98ce" path="/var/lib/kubelet/pods/fb7d6160-b41b-400b-9316-35478bbe98ce/volumes" Nov 03 16:46:08.054350 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.054318 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb"] Nov 03 16:46:08.055222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055197 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8f531f1d-6b44-4499-b1c2-cac8a28ef78d" containerName="maas-api" Nov 03 16:46:08.055222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055222 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f531f1d-6b44-4499-b1c2-cac8a28ef78d" containerName="maas-api" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055241 4539 cpu_manager.go:414] "containerMap: removing container" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055246 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055254 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055259 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055271 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055276 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055286 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055290 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055300 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055304 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055315 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055319 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055326 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055331 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055337 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055341 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055354 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055358 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055363 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055368 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055377 4539 cpu_manager.go:414] "containerMap: removing container" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055382 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055388 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055392 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055405 4539 cpu_manager.go:414] "containerMap: removing container" podUID="fb7d6160-b41b-400b-9316-35478bbe98ce" containerName="maas-api" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055410 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7d6160-b41b-400b-9316-35478bbe98ce" containerName="maas-api" Nov 03 16:46:08.055406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055416 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055437 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055452 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055462 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055549 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055554 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055560 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055565 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f4ba74-0cc1-4ab0-a165-c522537165da" containerName="istio-proxy" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055573 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055579 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7d6160-b41b-400b-9316-35478bbe98ce" containerName="maas-api" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055585 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055591 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055596 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055601 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f531f1d-6b44-4499-b1c2-cac8a28ef78d" containerName="maas-api" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055606 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055612 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055618 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055623 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:08.056214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.055628 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:08.060223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.060207 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.076336 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.076316 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-98e3b9-serving-cert\"" Nov 03 16:46:08.087514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.087490 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb"] Nov 03 16:46:08.188549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.188519 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:08.249975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.249935 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.250134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.250089 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.250134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.250114 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkq6\" (UniqueName: \"kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.250247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.250159 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.336991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.335129 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:08.352030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.351998 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.352202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.352074 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.352202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.352090 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkq6\" (UniqueName: \"kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.352202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.352130 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.352748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.352723 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.354861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.354838 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.354945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.354838 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.377895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.377857 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkq6\" (UniqueName: \"kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6\") pod \"maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.673149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.673115 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:08.808082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.807727 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:46:08.863967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.863937 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:46:08.869873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.869849 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:46:08.962485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962415 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.962670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962523 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.962670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962555 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.962670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962590 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.962670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962642 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962731 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962764 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.962898 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963074 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963045 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.963340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963093 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.963340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963126 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.963340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963162 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963196 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963229 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963281 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.963737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963349 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo\") pod \"37fcd313-2648-4deb-aa60-364a21a54097\" (UID: \"37fcd313-2648-4deb-aa60-364a21a54097\") " Nov 03 16:46:08.963737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963389 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.963737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963494 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo\") pod \"fa033686-2d07-4941-9839-1e2b6d759464\" (UID: \"fa033686-2d07-4941-9839-1e2b6d759464\") " Nov 03 16:46:08.964007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963787 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.964007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.963989 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.964200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.964173 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.964361 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.964687 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965009 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965326 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965370 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965492 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965535 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965861 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965882 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965896 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965909 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965922 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965933 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa033686-2d07-4941-9839-1e2b6d759464-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965944 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fcd313-2648-4deb-aa60-364a21a54097-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965954 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965978 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.966002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.965991 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.970455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.970398 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "kube-api-access-rzkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:08.970951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.970925 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.971068 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.971045 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:08.971241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.971224 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:08.971293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.971240 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:08.973750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.973729 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:08.973837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.973810 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo" pod "37fcd313-2648-4deb-aa60-364a21a54097" (UID: "37fcd313-2648-4deb-aa60-364a21a54097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:08.974113 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.974095 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz" pod "fa033686-2d07-4941-9839-1e2b6d759464" (UID: "fa033686-2d07-4941-9839-1e2b6d759464"). InnerVolumeSpecName "kube-api-access-8thrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:08.989825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.989792 4539 generic.go:356] "Generic (PLEG): container finished" podID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerID="d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac" exitCode=0 Nov 03 16:46:08.989953 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.989932 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" event={"ID":"a2fe11d1-873c-468f-980a-2787c5b2cdd6","Type":"ContainerDied","Data":"d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac"} Nov 03 16:46:08.990005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.989992 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" event={"ID":"a2fe11d1-873c-468f-980a-2787c5b2cdd6","Type":"ContainerDied","Data":"c955726c7ff25e7baa1228c6facc4f3203998942854ca4d5e6dd3544b338d289"} Nov 03 16:46:08.990050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.990026 4539 scope.go:122] "RemoveContainer" containerID="d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac" Nov 03 16:46:08.990181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.990157 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg" Nov 03 16:46:08.992349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.992324 4539 generic.go:356] "Generic (PLEG): container finished" podID="fa033686-2d07-4941-9839-1e2b6d759464" containerID="c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636" exitCode=0 Nov 03 16:46:08.992555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.992538 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" Nov 03 16:46:08.993142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.993123 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" event={"ID":"fa033686-2d07-4941-9839-1e2b6d759464","Type":"ContainerDied","Data":"c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636"} Nov 03 16:46:08.993216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.993182 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg" event={"ID":"fa033686-2d07-4941-9839-1e2b6d759464","Type":"ContainerDied","Data":"c38f17447350aaee042a7f1a063f6ebaf2a91f80d214b13e4c4ad579babdb025"} Nov 03 16:46:08.995416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.995396 4539 generic.go:356] "Generic (PLEG): container finished" podID="37fcd313-2648-4deb-aa60-364a21a54097" containerID="93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d" exitCode=0 Nov 03 16:46:08.995546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.995530 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" event={"ID":"37fcd313-2648-4deb-aa60-364a21a54097","Type":"ContainerDied","Data":"93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d"} Nov 03 16:46:08.995607 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.995570 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" event={"ID":"37fcd313-2648-4deb-aa60-364a21a54097","Type":"ContainerDied","Data":"c1bae59a44a169c91f7e62c7d425726a2a78a88ead7f7aff0ccffe87c5597da7"} Nov 03 16:46:08.995676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:08.995663 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k" Nov 03 16:46:09.017928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.017899 4539 scope.go:122] "RemoveContainer" containerID="d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac" Nov 03 16:46:09.019502 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:46:09.018395 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac\": container with ID starting with d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac not found: ID does not exist" containerID="d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac" Nov 03 16:46:09.019590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.019520 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac"} err="failed to get container status \"d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac\": rpc error: code = NotFound desc = could not find container \"d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac\": container with ID starting with d2da75de6a2b2debdb2bc5701f68c15305bfc36426bdf9e8e9b4691acf7432ac not found: ID does not exist" Nov 03 16:46:09.019590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.019547 4539 scope.go:122] "RemoveContainer" containerID="c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636" Nov 03 16:46:09.022681 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:46:09.022653 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdac765b_0749_4767_a4c7_97adc4eda6d6.slice/crio-d8fb8c07ffa3e021682bce60ff8a6fcf1e52b38e51707370a1d812dd89c16233 WatchSource:0}: Error finding container d8fb8c07ffa3e021682bce60ff8a6fcf1e52b38e51707370a1d812dd89c16233: Status 404 returned error can't find the container with id d8fb8c07ffa3e021682bce60ff8a6fcf1e52b38e51707370a1d812dd89c16233 Nov 03 16:46:09.024258 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.024239 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:46:09.027490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.027465 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb"] Nov 03 16:46:09.032143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.032125 4539 scope.go:122] "RemoveContainer" containerID="c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636" Nov 03 16:46:09.032393 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:46:09.032373 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636\": container with ID starting with c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636 not found: ID does not exist" containerID="c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636" Nov 03 16:46:09.032524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.032401 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636"} err="failed to get container status \"c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636\": rpc error: code = NotFound desc = could not find container \"c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636\": container with ID starting with c1d2ba895e321fb02424a0b0f03deb36edbe80350ad7807d0a9fea0c382d6636 not found: ID does not exist" Nov 03 16:46:09.032524 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.032418 4539 scope.go:122] "RemoveContainer" containerID="93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d" Nov 03 16:46:09.044677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.044655 4539 scope.go:122] "RemoveContainer" containerID="93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d" Nov 03 16:46:09.044994 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:46:09.044971 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d\": container with ID starting with 93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d not found: ID does not exist" containerID="93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d" Nov 03 16:46:09.045073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.045006 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d"} err="failed to get container status \"93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d\": rpc error: code = NotFound desc = could not find container \"93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d\": container with ID starting with 93d7722ddae4ab142b1be732339eae375858cb51c0fc5d6e1aa6cd3eb67eb95d not found: ID does not exist" Nov 03 16:46:09.066712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.066683 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.066767 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.066870 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.066916 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.066938 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.066978 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067020 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067087 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067194 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s\") pod \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\" (UID: \"a2fe11d1-873c-468f-980a-2787c5b2cdd6\") " Nov 03 16:46:09.067495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067220 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:09.067495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067235 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:09.067495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067235 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:09.067495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067405 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067504 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067821 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067841 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067854 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thrz\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-kube-api-access-8thrz\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067863 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067870 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067878 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fcd313-2648-4deb-aa60-364a21a54097-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067887 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067895 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa033686-2d07-4941-9839-1e2b6d759464-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.067896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067903 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa033686-2d07-4941-9839-1e2b6d759464-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.068380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067913 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fcd313-2648-4deb-aa60-364a21a54097-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.068380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067927 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa033686-2d07-4941-9839-1e2b6d759464-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.068380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067934 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzkx9\" (UniqueName: \"kubernetes.io/projected/37fcd313-2648-4deb-aa60-364a21a54097-kube-api-access-rzkx9\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.068380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.067942 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.069667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.069642 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:09.069871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.069845 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:09.070090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.070067 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:09.078560 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.078482 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s" pod "a2fe11d1-873c-468f-980a-2787c5b2cdd6" (UID: "a2fe11d1-873c-468f-980a-2787c5b2cdd6"). InnerVolumeSpecName "kube-api-access-56g2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:09.083121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.083096 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg"] Nov 03 16:46:09.086542 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.086519 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-kt9cg"] Nov 03 16:46:09.116798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.116723 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k"] Nov 03 16:46:09.123388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.123359 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-5v22k"] Nov 03 16:46:09.168911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.168878 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.168911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.168907 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.168911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.168917 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56g2s\" (UniqueName: \"kubernetes.io/projected/a2fe11d1-873c-468f-980a-2787c5b2cdd6-kube-api-access-56g2s\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.169218 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.168929 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fe11d1-873c-468f-980a-2787c5b2cdd6-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:09.344352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.344322 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg"] Nov 03 16:46:09.350099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.350073 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-xzdbg"] Nov 03 16:46:09.434770 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.434696 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:09.606626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:09.606569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:10.007457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.005855 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" event={"ID":"fdac765b-0749-4767-a4c7-97adc4eda6d6","Type":"ContainerStarted","Data":"fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843"} Nov 03 16:46:10.007457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.005924 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:10.007457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.005947 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" event={"ID":"fdac765b-0749-4767-a4c7-97adc4eda6d6","Type":"ContainerStarted","Data":"d8fb8c07ffa3e021682bce60ff8a6fcf1e52b38e51707370a1d812dd89c16233"} Nov 03 16:46:10.047090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.047032 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" podStartSLOduration=1.587959498 podStartE2EDuration="2.047015344s" totalImagesPullingTime="459.055846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:10.046380712 +0000 UTC m=+1913.739651735" watchObservedRunningTime="2026-11-03 16:46:10.047015344 +0000 UTC m=+1913.740286366" Nov 03 16:46:10.469083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.469046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:10.469716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.469694 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fcd313-2648-4deb-aa60-364a21a54097" path="/var/lib/kubelet/pods/37fcd313-2648-4deb-aa60-364a21a54097/volumes" Nov 03 16:46:10.470222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.470207 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" path="/var/lib/kubelet/pods/a2fe11d1-873c-468f-980a-2787c5b2cdd6/volumes" Nov 03 16:46:10.470678 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.470665 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa033686-2d07-4941-9839-1e2b6d759464" path="/var/lib/kubelet/pods/fa033686-2d07-4941-9839-1e2b6d759464/volumes" Nov 03 16:46:10.867529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.867499 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz"] Nov 03 16:46:10.867854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.867719 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" containerID="cri-o://967dd21b2da893055df769acb6b642fdf7b333e416f9d3741ea5213e52ecf8df" gracePeriod=30 Nov 03 16:46:10.963993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:10.963952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:11.110884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:11.110839 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:11.258876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:11.258785 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:11.797061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:11.797034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:12.166000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:12.165914 4539 patch_prober.go:29] interesting pod/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:46:12.166154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:12.165991 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:46:13.669887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:13.669856 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:13.812613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:13.812585 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:14.014644 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:14.014558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:15.679307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:15.679278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:16.023687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.023655 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:46:16.084006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.081494 4539 generic.go:356] "Generic (PLEG): container finished" podID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerID="967dd21b2da893055df769acb6b642fdf7b333e416f9d3741ea5213e52ecf8df" exitCode=0 Nov 03 16:46:16.084006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.081641 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" event={"ID":"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5","Type":"ContainerDied","Data":"967dd21b2da893055df769acb6b642fdf7b333e416f9d3741ea5213e52ecf8df"} Nov 03 16:46:16.084006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.081702 4539 scope.go:122] "RemoveContainer" containerID="7d3e69ab48de4e7f90a6d5625e5a6a86eec737e5b726ec72db3c7401ec26f98b" Nov 03 16:46:16.189113 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.183221 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:46:16.269795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.269695 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.269795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.269777 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.269821 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.269876 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.269915 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.269975 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270111 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:16.270244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270186 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270233 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270285 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy\") pod \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\" (UID: \"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5\") " Nov 03 16:46:16.270402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270297 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:16.270901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270837 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.270901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.270868 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.271616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.271591 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:16.271805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.271783 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:16.273302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.273264 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:16.273588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.273563 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "kube-api-access-gms2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:16.273694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.273618 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:16.273756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.273707 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:16.273937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.273914 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy" pod "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" (UID: "30cbb72b-92ba-4c9c-bc76-ceb86906a0b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:16.372332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372296 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.372332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372335 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.372634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372350 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.372634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372365 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gms2b\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-kube-api-access-gms2b\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.372634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372383 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.372634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372398 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.372634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.372411 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:16.728748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.728708 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q"] Nov 03 16:46:16.729496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.728956 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" containerID="cri-o://ff8aff3444b27de0da89270b300769b3477f4ca661fcd9e78fadc1c8d978132f" gracePeriod=30 Nov 03 16:46:16.734345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.734320 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn"] Nov 03 16:46:16.734576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.734541 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" containerID="cri-o://774e0c3ed3b6a63973f28d0226c6d62a940ff5831d7a1482cec715bab2e1c908" gracePeriod=30 Nov 03 16:46:16.740459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.740413 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg"] Nov 03 16:46:16.740701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.740659 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" containerID="cri-o://b8550d882900a5a408bb16521dbbbb45e478fcfc6be45a49aa6a9eb1cf11c6b6" gracePeriod=30 Nov 03 16:46:16.927040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:16.927006 4539 scope.go:122] "RemoveContainer" containerID="c13fbe0d586b130bd9f48284c804577bd085106e58ee65abdb13c3e710a47556" Nov 03 16:46:17.089529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:17.089498 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" Nov 03 16:46:17.089712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:17.089527 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz" event={"ID":"30cbb72b-92ba-4c9c-bc76-ceb86906a0b5","Type":"ContainerDied","Data":"3eb3be91d504c8f103be4b6b6df1a9a1b76df1b11ecb515f52f4daf46f49f275"} Nov 03 16:46:17.089712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:17.089607 4539 scope.go:122] "RemoveContainer" containerID="967dd21b2da893055df769acb6b642fdf7b333e416f9d3741ea5213e52ecf8df" Nov 03 16:46:17.147516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:17.147485 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz"] Nov 03 16:46:17.155354 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:17.155325 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-db3c6b-openshift-default-dc96c888f-9k7gz"] Nov 03 16:46:17.525897 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:17.525842 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:18.446945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:18.446909 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:18.469902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:18.469877 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" path="/var/lib/kubelet/pods/30cbb72b-92ba-4c9c-bc76-ceb86906a0b5/volumes" Nov 03 16:46:18.613664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:18.613633 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:19.585882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:19.585802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:19.771273 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:19.771226 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:21.345067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.345025 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:21.383736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.383707 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq"] Nov 03 16:46:21.384318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384300 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:21.384318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384318 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384335 4539 cpu_manager.go:414] "containerMap: removing container" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384340 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384354 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384362 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384373 4539 cpu_manager.go:414] "containerMap: removing container" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384378 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384388 4539 cpu_manager.go:414] "containerMap: removing container" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384393 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384405 4539 cpu_manager.go:414] "containerMap: removing container" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384410 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384442 4539 cpu_manager.go:414] "containerMap: removing container" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384449 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131eb6da-b0bf-49df-9a3f-33480f237df0" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384468 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" Nov 03 16:46:21.384527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384473 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e27eaf5-a18a-4bef-b46b-aa4ac77ee326" containerName="istio-proxy" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384580 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384596 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea5d4f9-0a5d-4949-8fa7-ddd4302e6ee8" containerName="cleanup" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384629 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fe11d1-873c-468f-980a-2787c5b2cdd6" containerName="istio-proxy" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384639 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fcd313-2648-4deb-aa60-364a21a54097" containerName="istio-proxy" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384647 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee93ceb4-1fff-42be-9345-864b01adb451" containerName="cleanup" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384654 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2a9fec-519d-4761-9056-e4da2c2528ae" containerName="cleanup" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384671 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf2b81f-db47-4153-bee2-9c148ff40f59" containerName="cleanup" Nov 03 16:46:21.384952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.384676 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa033686-2d07-4941-9839-1e2b6d759464" containerName="istio-proxy" Nov 03 16:46:21.389968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.389950 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.431651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.431620 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-624649-serving-cert\"" Nov 03 16:46:21.453464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.445551 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq"] Nov 03 16:46:21.457512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.457490 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.457637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.457617 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.457763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.457645 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.457942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.457920 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmnh\" (UniqueName: \"kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.559186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.559155 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmnh\" (UniqueName: \"kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.559369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.559226 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.559369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.559295 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.559369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.559318 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.559977 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.559952 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.561800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.561782 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.561896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.561877 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.590406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.590377 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmnh\" (UniqueName: \"kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh\") pod \"maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.600979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.600915 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:21.703931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.703901 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:21.983122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.983083 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn"] Nov 03 16:46:21.984275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.984246 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cbb72b-92ba-4c9c-bc76-ceb86906a0b5" containerName="istio-proxy" Nov 03 16:46:21.991829 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:21.991802 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.051480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.051330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:22.068448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.053823 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e0a112-openshift-default-dockercfg-2cjpt\"" Nov 03 16:46:22.068448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.067615 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn"] Nov 03 16:46:22.069500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.069471 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt75p\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.069761 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.069743 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.069902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.069889 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.070022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.070009 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.070134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.070122 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.070277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.070264 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.070550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.070530 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.070652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.070638 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.070819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.070806 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.128923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.128842 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq"] Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176273 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176333 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176360 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176479 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176504 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt75p\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176623 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176699 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176756 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.176801 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.177610 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.177867 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.178108 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.178812 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.179498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.179358 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.198033 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.197932 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.198924 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.198907 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.215702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.215644 4539 generic.go:356] "Generic (PLEG): container finished" podID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerID="b8550d882900a5a408bb16521dbbbb45e478fcfc6be45a49aa6a9eb1cf11c6b6" exitCode=0 Nov 03 16:46:22.216067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.216039 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" event={"ID":"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861","Type":"ContainerDied","Data":"b8550d882900a5a408bb16521dbbbb45e478fcfc6be45a49aa6a9eb1cf11c6b6"} Nov 03 16:46:22.233446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.233380 4539 generic.go:356] "Generic (PLEG): container finished" podID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerID="ff8aff3444b27de0da89270b300769b3477f4ca661fcd9e78fadc1c8d978132f" exitCode=0 Nov 03 16:46:22.233998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.233678 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" event={"ID":"529563fd-3eba-42b4-9f15-bae338d84fd7","Type":"ContainerDied","Data":"ff8aff3444b27de0da89270b300769b3477f4ca661fcd9e78fadc1c8d978132f"} Nov 03 16:46:22.237381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.237181 4539 generic.go:356] "Generic (PLEG): container finished" podID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerID="774e0c3ed3b6a63973f28d0226c6d62a940ff5831d7a1482cec715bab2e1c908" exitCode=0 Nov 03 16:46:22.237381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.237311 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" event={"ID":"a041f70b-c4aa-43b0-8767-e9834c2aa6fc","Type":"ContainerDied","Data":"774e0c3ed3b6a63973f28d0226c6d62a940ff5831d7a1482cec715bab2e1c908"} Nov 03 16:46:22.293594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.293567 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.293960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.293937 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt75p\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p\") pod \"e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.345236 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.344944 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:22.392170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.392146 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:46:22.393006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.392804 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:22.434368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.434320 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.488870 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.488972 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489006 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489124 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489189 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489327 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489360 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489410 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489465 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket\") pod \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\" (UID: \"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861\") " Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489507 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.489724 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.490135 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.490156 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.490417 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:22.491124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.490654 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.494018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.493960 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.498874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.496317 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:22.498874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.497224 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:22.499513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.499476 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "kube-api-access-9t6xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:22.500771 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.500739 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy" pod "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" (UID: "d0133ecf-85d2-4d69-a6c2-abfe8b6cf861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.557710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.557679 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:46:22.591380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591333 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591462 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591509 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591656 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591730 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591788 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591914 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.591999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.591979 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.592066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592053 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo\") pod \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\" (UID: \"a041f70b-c4aa-43b0-8767-e9834c2aa6fc\") " Nov 03 16:46:22.592827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592803 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.592941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592833 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.592941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592849 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.592941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592864 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.592941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592879 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6xt\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-kube-api-access-9t6xt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.592941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592892 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.592941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.592904 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.594665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.594070 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.594665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.594074 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:22.594665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.594271 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.594665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.594637 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.594927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.594711 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.596598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.596563 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "kube-api-access-dcrq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:22.597880 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.597842 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:22.598661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.598635 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:22.601185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.601151 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy" pod "a041f70b-c4aa-43b0-8767-e9834c2aa6fc" (UID: "a041f70b-c4aa-43b0-8767-e9834c2aa6fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.625591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.625567 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:22.693844 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.693814 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.693957 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694026 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694055 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694147 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694203 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694228 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.694358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694302 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694436 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694358 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694449 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.694501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694458 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy\") pod \"529563fd-3eba-42b4-9f15-bae338d84fd7\" (UID: \"529563fd-3eba-42b4-9f15-bae338d84fd7\") " Nov 03 16:46:22.694600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694503 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:22.694676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694655 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.694958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.694889 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695083 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695104 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695117 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695128 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695139 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695151 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcrq7\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-kube-api-access-dcrq7\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695165 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529563fd-3eba-42b4-9f15-bae338d84fd7-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695177 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695191 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695214 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695227 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695240 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695252 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a041f70b-c4aa-43b0-8767-e9834c2aa6fc-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.696881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.695264 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.697720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.697573 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:22.697720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.697668 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "kube-api-access-vm5p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:22.698656 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.698630 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:22.699337 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.699178 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy" pod "529563fd-3eba-42b4-9f15-bae338d84fd7" (UID: "529563fd-3eba-42b4-9f15-bae338d84fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:22.772180 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.770032 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn"] Nov 03 16:46:22.772180 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:46:22.771130 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a9fad8_f0eb_405e_a3c8_25ca54b6a9e8.slice/crio-df6397e822ad91c500d67f05532e4e967cffc254e46f63dffc437c49938ee1aa WatchSource:0}: Error finding container df6397e822ad91c500d67f05532e4e967cffc254e46f63dffc437c49938ee1aa: Status 404 returned error can't find the container with id df6397e822ad91c500d67f05532e4e967cffc254e46f63dffc437c49938ee1aa Nov 03 16:46:22.774763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.774726 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:46:22.774890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.774801 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:46:22.774890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.774832 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:46:22.797408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.797338 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.797408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.797371 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.797408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.797385 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm5p4\" (UniqueName: \"kubernetes.io/projected/529563fd-3eba-42b4-9f15-bae338d84fd7-kube-api-access-vm5p4\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:22.797408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:22.797401 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529563fd-3eba-42b4-9f15-bae338d84fd7-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.257689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.256629 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" event={"ID":"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8","Type":"ContainerStarted","Data":"9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f"} Nov 03 16:46:23.257689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.256697 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" event={"ID":"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8","Type":"ContainerStarted","Data":"df6397e822ad91c500d67f05532e4e967cffc254e46f63dffc437c49938ee1aa"} Nov 03 16:46:23.261172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.260334 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" event={"ID":"529563fd-3eba-42b4-9f15-bae338d84fd7","Type":"ContainerDied","Data":"20d0dc3f1880813fa2e85ccebd728f8e2ef2da612216cf1a97ea456381083a43"} Nov 03 16:46:23.261172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.260391 4539 scope.go:122] "RemoveContainer" containerID="ff8aff3444b27de0da89270b300769b3477f4ca661fcd9e78fadc1c8d978132f" Nov 03 16:46:23.261172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.260529 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" Nov 03 16:46:23.286477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.281961 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" event={"ID":"a041f70b-c4aa-43b0-8767-e9834c2aa6fc","Type":"ContainerDied","Data":"aa17d54faf72be07b662e34da464fee2c4e55e375565102f2e8e964e3e701a1a"} Nov 03 16:46:23.286477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.283448 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn" Nov 03 16:46:23.322810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.322663 4539 scope.go:122] "RemoveContainer" containerID="774e0c3ed3b6a63973f28d0226c6d62a940ff5831d7a1482cec715bab2e1c908" Nov 03 16:46:23.325333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.325143 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" event={"ID":"f5fb30bb-b719-4f38-8649-7d51248da1bb","Type":"ContainerStarted","Data":"881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1"} Nov 03 16:46:23.325333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.325242 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" event={"ID":"f5fb30bb-b719-4f38-8649-7d51248da1bb","Type":"ContainerStarted","Data":"14150150586c5e274e2de23787cba9a761d5b6c9bd616bc9dc4e29b020025d8b"} Nov 03 16:46:23.326413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.326390 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:23.350517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.348116 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" event={"ID":"d0133ecf-85d2-4d69-a6c2-abfe8b6cf861","Type":"ContainerDied","Data":"6f4dc58aa9fd3ed331c9458c8db8669f0ce03ca77ffd1867b11da04137f8aa38"} Nov 03 16:46:23.350517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.348266 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg" Nov 03 16:46:23.366787 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.366740 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podStartSLOduration=2.366725739 podStartE2EDuration="2.366725739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:23.32925918 +0000 UTC m=+1927.022530203" watchObservedRunningTime="2026-11-03 16:46:23.366725739 +0000 UTC m=+1927.059996760" Nov 03 16:46:23.369986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.369961 4539 scope.go:122] "RemoveContainer" containerID="b8550d882900a5a408bb16521dbbbb45e478fcfc6be45a49aa6a9eb1cf11c6b6" Nov 03 16:46:23.378881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.378772 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" podStartSLOduration=2.000253929 podStartE2EDuration="2.378756645s" totalImagesPullingTime="378.502716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:23.374515178 +0000 UTC m=+1927.067786203" watchObservedRunningTime="2026-11-03 16:46:23.378756645 +0000 UTC m=+1927.072027671" Nov 03 16:46:23.395763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.394181 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:23.401089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.401041 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:23.434476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.434442 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q"] Nov 03 16:46:23.440416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.440389 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q"] Nov 03 16:46:23.486462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.486419 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg"] Nov 03 16:46:23.500145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.500110 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-nrpwg"] Nov 03 16:46:23.502029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.501966 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:46:23.502814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.502777 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wwp5q" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:46:23.544605 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.544568 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn"] Nov 03 16:46:23.550978 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:23.550937 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-qswsn"] Nov 03 16:46:24.112180 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:24.112148 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:24.359017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:24.358987 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:24.359965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:24.359946 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:46:24.469859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:24.469775 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" path="/var/lib/kubelet/pods/529563fd-3eba-42b4-9f15-bae338d84fd7/volumes" Nov 03 16:46:24.470490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:24.470470 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" path="/var/lib/kubelet/pods/a041f70b-c4aa-43b0-8767-e9834c2aa6fc/volumes" Nov 03 16:46:24.471121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:24.471102 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" path="/var/lib/kubelet/pods/d0133ecf-85d2-4d69-a6c2-abfe8b6cf861/volumes" Nov 03 16:46:25.978202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:25.978171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:26.214639 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:46:26.214599 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:46:27.530178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:27.530141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:27.711106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:27.711074 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:27.903694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:27.903584 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:28.095894 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:28.095866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:30.185954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:30.185870 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:30.371677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:30.371649 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:46:32.465552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:32.465512 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:33.658805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:33.658772 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:33.800451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:33.800385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:34.987331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:34.987296 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:37.052328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:37.052296 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:39.184873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:39.184841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:39.854136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:39.854105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:40.240167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.240040 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q"] Nov 03 16:46:40.241847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.241816 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" Nov 03 16:46:40.241847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.241849 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" Nov 03 16:46:40.242090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.241868 4539 cpu_manager.go:414] "containerMap: removing container" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" Nov 03 16:46:40.242090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.241877 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" Nov 03 16:46:40.242090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.241923 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" Nov 03 16:46:40.242090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.241933 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" Nov 03 16:46:40.242311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.242135 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0133ecf-85d2-4d69-a6c2-abfe8b6cf861" containerName="istio-proxy" Nov 03 16:46:40.242311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.242152 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="529563fd-3eba-42b4-9f15-bae338d84fd7" containerName="istio-proxy" Nov 03 16:46:40.242311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.242171 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a041f70b-c4aa-43b0-8767-e9834c2aa6fc" containerName="istio-proxy" Nov 03 16:46:40.255753 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.255716 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.275617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.275587 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q"] Nov 03 16:46:40.277662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.277635 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-87b132-openshift-default-dockercfg-wzhn5\"" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.299960 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300051 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzj7\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300209 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300273 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300325 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300368 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300447 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300717 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.307465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.300776 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.401964 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzj7\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402029 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402084 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402122 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402161 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402246 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402212 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402572 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402327 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402572 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402394 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402570 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402880 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402848 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.402980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402924 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.403060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.402987 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.403060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.403036 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.403220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.403199 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.404383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.404361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:40.405121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.405098 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.405222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.405146 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.450630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.430328 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzj7\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.450630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.449417 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token\") pod \"e2e-api-b-87b132-openshift-default-546b679549-rmj5q\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.620485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.597659 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:40.667227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.667048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:40.861005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:40.860966 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:41.088795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.085684 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:46:41.088795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.085777 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:46:41.088795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.085821 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:46:41.088795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.086134 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q"] Nov 03 16:46:41.557105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.557070 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:41.673298 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.673257 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" event={"ID":"d6965863-e14b-43ed-afcb-f1b8c24caaa9","Type":"ContainerStarted","Data":"e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640"} Nov 03 16:46:41.673507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.673321 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" event={"ID":"d6965863-e14b-43ed-afcb-f1b8c24caaa9","Type":"ContainerStarted","Data":"155b7ceffa8e0bce9361d90b2e26bbe72077a19fa363512e58a9a60f09314324"} Nov 03 16:46:41.737819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.737757 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podStartSLOduration=1.737739796 podStartE2EDuration="1.737739796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:41.727217753 +0000 UTC m=+1945.420488776" watchObservedRunningTime="2026-11-03 16:46:41.737739796 +0000 UTC m=+1945.431010818" Nov 03 16:46:41.853266 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:41.853180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:42.014972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:42.014921 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:42.601851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:42.601821 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:42.607533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:42.607510 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:42.652035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:42.652003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:42.693483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:42.693412 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:42.702979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:42.698690 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:46:43.064468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:43.062090 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:43.866157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:43.866114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:44.061165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:44.061132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:44.412463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:44.412417 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:44.629265 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:44.629234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:44.812073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:44.811985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:45.718635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:45.718603 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:46.211243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:46.211208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:46.385856 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:46.385807 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:48.254010 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:48.253974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:48.451450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:48.448362 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:48.911162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:48.911118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:49.782019 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:49.781990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:50.650832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.650801 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p"] Nov 03 16:46:50.655629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.655601 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.669792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.669762 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:50.672760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.672734 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e0a112-serving-cert\"" Nov 03 16:46:50.688604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.688575 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p"] Nov 03 16:46:50.816606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.816563 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhwk\" (UniqueName: \"kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.817100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.816636 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.817100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.816809 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.817100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.816868 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.844115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.844067 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:50.941547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.939382 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhwk\" (UniqueName: \"kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.945569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.945511 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.945855 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.945655 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.945855 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.945711 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.947508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.947145 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.992495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.981986 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:50.992495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:50.991738 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:51.046451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:51.020277 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:51.108444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:51.108390 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhwk\" (UniqueName: \"kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk\") pod \"maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:51.179491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:51.179454 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:51.274966 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:51.274930 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:51.603587 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:46:51.603552 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c50412e_f78a_40da_87f2_ac72644f1f17.slice/crio-9f760390108b7260784f808b1519c03065a8a861a7e44a12b98373319a3bda4e WatchSource:0}: Error finding container 9f760390108b7260784f808b1519c03065a8a861a7e44a12b98373319a3bda4e: Status 404 returned error can't find the container with id 9f760390108b7260784f808b1519c03065a8a861a7e44a12b98373319a3bda4e Nov 03 16:46:51.607290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:51.607188 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p"] Nov 03 16:46:51.776413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:51.776372 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" event={"ID":"9c50412e-f78a-40da-87f2-ac72644f1f17","Type":"ContainerStarted","Data":"9f760390108b7260784f808b1519c03065a8a861a7e44a12b98373319a3bda4e"} Nov 03 16:46:52.538073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:52.538034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:52.682048 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:52.682013 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:52.786310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:52.786277 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" event={"ID":"9c50412e-f78a-40da-87f2-ac72644f1f17","Type":"ContainerStarted","Data":"c9ab10e4a682d8e50ec3fd3b5665ea5ef2af7d82d564975df204bb1e16a01449"} Nov 03 16:46:52.786569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:52.786542 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:52.846743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:52.846649 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:52.847122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:52.847080 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" podStartSLOduration=2.5168615819999998 podStartE2EDuration="2.847068004s" totalImagesPullingTime="330.206422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:52.835780408 +0000 UTC m=+1956.529051431" watchObservedRunningTime="2026-11-03 16:46:52.847068004 +0000 UTC m=+1956.540339027" Nov 03 16:46:54.644868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:54.644837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:56.158304 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:46:56.158267 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:46:56.621694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:56.621654 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:58.397773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:58.397732 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:46:58.800059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:58.800020 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:46:58.911221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:46:58.911175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:00.969678 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:00.969643 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:02.311625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:02.311588 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:02.512292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:02.512255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:02.661920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:02.661788 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:03.200356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:03.200320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:04.912856 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:04.912812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:05.271203 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:05.271110 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:05.454067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:05.454034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:06.062034 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:06.061998 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:06.394335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:06.394245 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc"] Nov 03 16:47:06.394585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:06.394558 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" containerID="cri-o://212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70" gracePeriod=30 Nov 03 16:47:06.461171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:06.461134 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:07.001622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.001588 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7"] Nov 03 16:47:07.007017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.006996 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.021930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.021907 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-87b132-serving-cert\"" Nov 03 16:47:07.045301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.045263 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7"] Nov 03 16:47:07.121749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.121714 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.122146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.121896 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.122146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.122014 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxm4\" (UniqueName: \"kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.122146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.122124 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.146406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.146371 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:07.224367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.224316 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.224688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.224462 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxm4\" (UniqueName: \"kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.224688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.224600 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.224801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.224694 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.239539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.239495 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.243396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.243328 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.243877 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.243849 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.262418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.262331 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxm4\" (UniqueName: \"kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4\") pod \"maas-api-e2e-api-b-87b132-86786c655c-z72n7\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.293839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.293804 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:07.322270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.322225 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:07.439926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.439895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:07.632358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.632328 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7"] Nov 03 16:47:07.633387 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:47:07.633346 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6f3cc8_fecb_4f11_9e06_5c3f24001984.slice/crio-6b0bdf1dc559c442f115e05335990d31ce2548c6d1e6764831152ca334f78961 WatchSource:0}: Error finding container 6b0bdf1dc559c442f115e05335990d31ce2548c6d1e6764831152ca334f78961: Status 404 returned error can't find the container with id 6b0bdf1dc559c442f115e05335990d31ce2548c6d1e6764831152ca334f78961 Nov 03 16:47:07.919906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:07.919800 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" event={"ID":"dd6f3cc8-fecb-4f11-9e06-5c3f24001984","Type":"ContainerStarted","Data":"6b0bdf1dc559c442f115e05335990d31ce2548c6d1e6764831152ca334f78961"} Nov 03 16:47:08.930606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:08.930566 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" event={"ID":"dd6f3cc8-fecb-4f11-9e06-5c3f24001984","Type":"ContainerStarted","Data":"6fb29d40fd39e588eef2c541fb80ffc380342269d712c4d32563a3b36c821de0"} Nov 03 16:47:08.931148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:08.930766 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:08.974437 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:08.974372 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" podStartSLOduration=2.6398431159999998 podStartE2EDuration="2.974355431s" totalImagesPullingTime="334.512315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:08.971832904 +0000 UTC m=+1972.665103973" watchObservedRunningTime="2026-11-03 16:47:08.974355431 +0000 UTC m=+1972.667626452" Nov 03 16:47:09.262122 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:09.262035 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:11.125757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.125718 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:11.285484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.285454 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:11.682989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.682916 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:47:11.791432 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791397 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791512 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791590 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791652 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791694 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791770 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791831 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.791969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791950 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.792038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.791986 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket\") pod \"a7c42f87-5b3f-4724-81b5-18be65528ec0\" (UID: \"a7c42f87-5b3f-4724-81b5-18be65528ec0\") " Nov 03 16:47:11.792038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792015 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:11.792170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792122 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:11.792257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792236 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:11.792346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792329 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:11.792587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792278 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:11.792587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792548 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.792587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792560 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.792587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792568 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c42f87-5b3f-4724-81b5-18be65528ec0-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.792587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792578 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.792872 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.792589 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.794082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.794053 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "kube-api-access-rpqc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:11.794082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.794056 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:47:11.794269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.794094 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:11.794331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.794285 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy" pod "a7c42f87-5b3f-4724-81b5-18be65528ec0" (UID: "a7c42f87-5b3f-4724-81b5-18be65528ec0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:11.893598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.893559 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.893598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.893593 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.893598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.893603 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqc6\" (UniqueName: \"kubernetes.io/projected/a7c42f87-5b3f-4724-81b5-18be65528ec0-kube-api-access-rpqc6\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.893856 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.893614 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c42f87-5b3f-4724-81b5-18be65528ec0-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:11.954332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.954234 4539 generic.go:356] "Generic (PLEG): container finished" podID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerID="212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70" exitCode=0 Nov 03 16:47:11.954332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.954300 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" Nov 03 16:47:11.954332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.954312 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" event={"ID":"a7c42f87-5b3f-4724-81b5-18be65528ec0","Type":"ContainerDied","Data":"212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70"} Nov 03 16:47:11.954651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.954368 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc" event={"ID":"a7c42f87-5b3f-4724-81b5-18be65528ec0","Type":"ContainerDied","Data":"e385a227431ea661b41aefd387880f1d5709d4d75420bb485449116f31c34f0f"} Nov 03 16:47:11.954651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.954389 4539 scope.go:122] "RemoveContainer" containerID="212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70" Nov 03 16:47:11.965283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.965263 4539 scope.go:122] "RemoveContainer" containerID="c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f" Nov 03 16:47:11.975060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.975037 4539 scope.go:122] "RemoveContainer" containerID="212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70" Nov 03 16:47:11.975372 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:11.975350 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70\": container with ID starting with 212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70 not found: ID does not exist" containerID="212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70" Nov 03 16:47:11.975444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.975382 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70"} err="failed to get container status \"212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70\": rpc error: code = NotFound desc = could not find container \"212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70\": container with ID starting with 212d62245fa6d36ab6b8cf16cc53f9cc45d976c1012143a424b4a183c3288d70 not found: ID does not exist" Nov 03 16:47:11.975444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.975400 4539 scope.go:122] "RemoveContainer" containerID="c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f" Nov 03 16:47:11.975694 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:11.975679 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f\": container with ID starting with c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f not found: ID does not exist" containerID="c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f" Nov 03 16:47:11.975745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:11.975697 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f"} err="failed to get container status \"c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f\": rpc error: code = NotFound desc = could not find container \"c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f\": container with ID starting with c34b403e99d2347a5cd9e618746f18577ab48f924d3dc24b385f9b48e7ed1a3f not found: ID does not exist" Nov 03 16:47:12.004943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:12.004865 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc"] Nov 03 16:47:12.013809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:12.013746 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-156c2a-openshift-default-669f4bd55-7cjbc"] Nov 03 16:47:12.329679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:12.329642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:12.469546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:12.469514 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" path="/var/lib/kubelet/pods/a7c42f87-5b3f-4724-81b5-18be65528ec0/volumes" Nov 03 16:47:12.530659 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:12.530624 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:12.820339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:12.820300 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:13.305531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.305491 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l"] Nov 03 16:47:13.306250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.306226 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" Nov 03 16:47:13.306250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.306250 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" Nov 03 16:47:13.306477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.306270 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" Nov 03 16:47:13.306477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.306276 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" Nov 03 16:47:13.306477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.306365 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" Nov 03 16:47:13.306477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.306374 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c42f87-5b3f-4724-81b5-18be65528ec0" containerName="istio-proxy" Nov 03 16:47:13.311395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.311369 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.338144 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.338107 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9947ab-serving-cert\"" Nov 03 16:47:13.351697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.351664 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l"] Nov 03 16:47:13.418241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.418207 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.418459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.418297 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.418459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.418354 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.418583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.418500 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7df\" (UniqueName: \"kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.442673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.442528 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:13.525263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.522033 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.525263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.522213 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.525263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.522379 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7df\" (UniqueName: \"kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.526604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.526555 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.526984 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:13.526913 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-serving-cert: secret "maas-api-e2e-ipp-a-9947ab-serving-cert" not found Nov 03 16:47:13.537683 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:13.537647 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls podName:8d4092e9-0202-4395-809c-fa9bad375cf3 nodeName:}" failed. No retries permitted until 2026-11-03 16:47:14.037605494 +0000 UTC m=+1977.730876504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls") pod "maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" (UID: "8d4092e9-0202-4395-809c-fa9bad375cf3") : secret "maas-api-e2e-ipp-a-9947ab-serving-cert" not found Nov 03 16:47:13.537912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.537774 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.542949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.542915 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.559576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.558828 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7df\" (UniqueName: \"kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:13.592139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.592106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:13.768103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:13.768067 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:14.044878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.044836 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:14.048321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.048278 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:14.229448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.225974 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:14.549538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.544123 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l"] Nov 03 16:47:14.947032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.947001 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:47:14.977390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.977351 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" event={"ID":"8d4092e9-0202-4395-809c-fa9bad375cf3","Type":"ContainerStarted","Data":"aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77"} Nov 03 16:47:14.977591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.977405 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" event={"ID":"8d4092e9-0202-4395-809c-fa9bad375cf3","Type":"ContainerStarted","Data":"86198c57319b38cc992b8fdaef0d1aba5b18fbddc69d2283b763fafda70494d3"} Nov 03 16:47:14.977591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:14.977448 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:15.034151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:15.034098 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" podStartSLOduration=1.7712861229999999 podStartE2EDuration="2.034082733s" totalImagesPullingTime="262.79661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:15.031228391 +0000 UTC m=+1978.724499417" watchObservedRunningTime="2026-11-03 16:47:15.034082733 +0000 UTC m=+1978.727353755" Nov 03 16:47:15.126692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:15.126580 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:16.962072 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:16.962025 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:17.022451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:17.019143 4539 scope.go:122] "RemoveContainer" containerID="5984e7ecd29ef0ddcf92ed159a14a79196494343e7e85d61f177cd00869bda6e" Nov 03 16:47:18.865598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:18.865567 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:19.034620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:19.034533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:20.597170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:20.597134 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:20.989135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:20.989057 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:47:21.986601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:21.986553 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:22.150004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:22.149974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:22.296139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:22.296079 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:22.500374 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:22.500335 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:24.546613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:24.546581 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:24.713605 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:24.713566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:24.857332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:24.857250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:25.241087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:25.240968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:25.399613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:25.399576 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:47:25.399812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:25.399671 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:47:25.513744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:25.513658 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:25.668242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:25.668209 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:26.162250 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:26.162217 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:47:26.247665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:26.247616 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:26.641991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:26.641920 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:27.170479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.170405 4539 generic.go:356] "Generic (PLEG): container finished" podID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerID="9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f" exitCode=0 Nov 03 16:47:27.170479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.170472 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" event={"ID":"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8","Type":"ContainerDied","Data":"9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f"} Nov 03 16:47:27.171002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.170904 4539 scope.go:122] "RemoveContainer" containerID="9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f" Nov 03 16:47:27.172657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.172631 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:47:27.172720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.172696 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:47:27.172756 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.172730 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:47:27.328390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:27.328354 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:28.178954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.178917 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" event={"ID":"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8","Type":"ContainerStarted","Data":"5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb"} Nov 03 16:47:28.232231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.232199 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:28.394135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.394096 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:47:28.394328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.394309 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:47:28.394380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.394355 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 16:47:28.777912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.777837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:28.951390 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:28.951356 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:29.105083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:29.104969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:29.247274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:29.247243 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:29.394414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:29.394316 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:47:29.394414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:29.394366 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 16:47:29.553702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:29.553656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:29.711388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:29.711291 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:30.394196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:30.394162 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:47:30.394913 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:30.394869 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 16:47:31.251313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:31.251281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:31.393542 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:31.393504 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:47:31.393736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:31.393565 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 16:47:32.394033 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:32.393996 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:47:32.394478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:32.394058 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 16:47:32.775511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:32.773091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:32.950406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:32.950366 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:33.157038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:33.156920 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:33.393695 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:33.393661 4539 patch_prober.go:29] interesting pod/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:47:33.393892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:33.393719 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 16:47:33.793314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:33.793280 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:33.967615 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:33.967579 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:34.190239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:34.190148 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:34.398574 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:34.398546 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:47:34.398922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:34.398904 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:47:34.399679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:34.399659 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:47:35.137732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:35.137686 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:35.293873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:35.293837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:35.440475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:35.440379 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:36.939151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:36.939108 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:37.235741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:37.235646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:37.449718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:37.449687 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:37.600996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:37.600961 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:39.256858 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:39.256828 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:41.154244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:41.154178 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:42.507674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:42.507642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:42.654540 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:42.654506 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:43.986312 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:43.986278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:44.364056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:44.364020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:44.519455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:44.519167 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:45.240288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:45.240254 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:45.399235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:45.399205 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:45.552080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:45.552033 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:46.146462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:46.145663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:46.302392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:46.302357 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:46.615975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:46.615915 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:48.089481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:48.089447 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:49.920045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:49.920005 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:50.048186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.048148 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb"] Nov 03 16:47:50.048724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.048687 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" podUID="fdac765b-0749-4767-a4c7-97adc4eda6d6" containerName="maas-api" containerID="cri-o://fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843" gracePeriod=30 Nov 03 16:47:50.090042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.090008 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:50.358065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.357699 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:50.565680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.565619 4539 generic.go:356] "Generic (PLEG): container finished" podID="fdac765b-0749-4767-a4c7-97adc4eda6d6" containerID="fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843" exitCode=0 Nov 03 16:47:50.568759 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.568415 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:47:50.569397 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.569372 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" event={"ID":"fdac765b-0749-4767-a4c7-97adc4eda6d6","Type":"ContainerDied","Data":"fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843"} Nov 03 16:47:50.569587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.569525 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" event={"ID":"fdac765b-0749-4767-a4c7-97adc4eda6d6","Type":"ContainerDied","Data":"d8fb8c07ffa3e021682bce60ff8a6fcf1e52b38e51707370a1d812dd89c16233"} Nov 03 16:47:50.569587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.569571 4539 scope.go:122] "RemoveContainer" containerID="fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843" Nov 03 16:47:50.582002 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.581895 4539 scope.go:122] "RemoveContainer" containerID="fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843" Nov 03 16:47:50.582394 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:50.582369 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843\": container with ID starting with fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843 not found: ID does not exist" containerID="fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843" Nov 03 16:47:50.582525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.582407 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843"} err="failed to get container status \"fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843\": rpc error: code = NotFound desc = could not find container \"fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843\": container with ID starting with fb13fec006000612044a9b522da530523068b212c9ff70570aa2734d57885843 not found: ID does not exist" Nov 03 16:47:50.704366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.703127 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca\") pod \"fdac765b-0749-4767-a4c7-97adc4eda6d6\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " Nov 03 16:47:50.704366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.703267 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs\") pod \"fdac765b-0749-4767-a4c7-97adc4eda6d6\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " Nov 03 16:47:50.704366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.703335 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls\") pod \"fdac765b-0749-4767-a4c7-97adc4eda6d6\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " Nov 03 16:47:50.704366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.703473 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6\" (UniqueName: \"kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6\") pod \"fdac765b-0749-4767-a4c7-97adc4eda6d6\" (UID: \"fdac765b-0749-4767-a4c7-97adc4eda6d6\") " Nov 03 16:47:50.705713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.705667 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca" pod "fdac765b-0749-4767-a4c7-97adc4eda6d6" (UID: "fdac765b-0749-4767-a4c7-97adc4eda6d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:50.711042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.710978 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6" pod "fdac765b-0749-4767-a4c7-97adc4eda6d6" (UID: "fdac765b-0749-4767-a4c7-97adc4eda6d6"). InnerVolumeSpecName "kube-api-access-ngkq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:50.712668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.712585 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls" pod "fdac765b-0749-4767-a4c7-97adc4eda6d6" (UID: "fdac765b-0749-4767-a4c7-97adc4eda6d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:50.731559 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.714307 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs" pod "fdac765b-0749-4767-a4c7-97adc4eda6d6" (UID: "fdac765b-0749-4767-a4c7-97adc4eda6d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:50.805204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.805168 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.805204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.805201 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdac765b-0749-4767-a4c7-97adc4eda6d6-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.805444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.805213 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkq6\" (UniqueName: \"kubernetes.io/projected/fdac765b-0749-4767-a4c7-97adc4eda6d6-kube-api-access-ngkq6\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.805444 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:50.805228 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdac765b-0749-4767-a4c7-97adc4eda6d6-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:47:51.198180 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.198144 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:51.352813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.352779 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:51.574619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.574586 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb" Nov 03 16:47:51.651992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.651949 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb"] Nov 03 16:47:51.663651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.663616 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98e3b9-c4dd8d759-j2gqb"] Nov 03 16:47:51.712239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.712103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:51.903336 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:51.903237 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:52.107207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:52.107170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:52.472148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:52.472094 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdac765b-0749-4767-a4c7-97adc4eda6d6" path="/var/lib/kubelet/pods/fdac765b-0749-4767-a4c7-97adc4eda6d6/volumes" Nov 03 16:47:52.487935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:52.487902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:53.205321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:53.205283 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:53.344106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:53.344069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:54.637876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.637841 4539 generic.go:356] "Generic (PLEG): container finished" podID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerID="e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640" exitCode=0 Nov 03 16:47:54.638463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.637925 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" event={"ID":"d6965863-e14b-43ed-afcb-f1b8c24caaa9","Type":"ContainerDied","Data":"e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640"} Nov 03 16:47:54.638535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.638493 4539 scope.go:122] "RemoveContainer" containerID="e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640" Nov 03 16:47:54.640241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.640214 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:47:54.640331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.640282 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:47:54.640331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.640318 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:47:54.888491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:54.888268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:55.647151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:55.647115 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" event={"ID":"d6965863-e14b-43ed-afcb-f1b8c24caaa9","Type":"ContainerStarted","Data":"e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae"} Nov 03 16:47:56.158247 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:47:56.158208 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:47:56.540692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.540658 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq"] Nov 03 16:47:56.541593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.541573 4539 cpu_manager.go:414] "containerMap: removing container" podUID="fdac765b-0749-4767-a4c7-97adc4eda6d6" containerName="maas-api" Nov 03 16:47:56.541691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.541596 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdac765b-0749-4767-a4c7-97adc4eda6d6" containerName="maas-api" Nov 03 16:47:56.541790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.541778 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdac765b-0749-4767-a4c7-97adc4eda6d6" containerName="maas-api" Nov 03 16:47:56.545831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.545782 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.569059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.568596 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b9e362-serving-cert\"" Nov 03 16:47:56.576555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.575645 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq"] Nov 03 16:47:56.626454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.605310 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:47:56.637569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.635549 4539 patch_prober.go:29] interesting pod/e2e-api-b-87b132-openshift-default-546b679549-rmj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 16:47:56.637569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.635626 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 16:47:56.659993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.659926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:56.736450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.735831 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.736450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.735965 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.736450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.736036 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczkd\" (UniqueName: \"kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.736450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.736176 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.821512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.815573 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:56.850457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.838217 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.850782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.850757 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczkd\" (UniqueName: \"kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.850995 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.850982 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.851130 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.851120 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.860482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.855362 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.863479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.863442 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.869347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.869311 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:56.879386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:56.879344 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczkd\" (UniqueName: \"kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd\") pod \"maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:57.168911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:57.168813 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:57.543866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:57.543833 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq"] Nov 03 16:47:57.545194 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:47:57.545162 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0641c0c8_4413_4dbc_a46e_8612fda4f05b.slice/crio-5a533598f3b50351e0e71a782c98a5f7d13dab1748f5fd370b14c31d2efdf14e WatchSource:0}: Error finding container 5a533598f3b50351e0e71a782c98a5f7d13dab1748f5fd370b14c31d2efdf14e: Status 404 returned error can't find the container with id 5a533598f3b50351e0e71a782c98a5f7d13dab1748f5fd370b14c31d2efdf14e Nov 03 16:47:57.607453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:57.604200 4539 patch_prober.go:29] interesting pod/e2e-api-b-87b132-openshift-default-546b679549-rmj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 16:47:57.607453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:57.604281 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 16:47:57.669945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:57.669904 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" event={"ID":"0641c0c8-4413-4dbc-a46e-8612fda4f05b","Type":"ContainerStarted","Data":"5a533598f3b50351e0e71a782c98a5f7d13dab1748f5fd370b14c31d2efdf14e"} Nov 03 16:47:57.693352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:57.693303 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:47:58.440475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.439811 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:58.586805 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.586715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:58.607497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.607458 4539 patch_prober.go:29] interesting pod/e2e-api-b-87b132-openshift-default-546b679549-rmj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 16:47:58.630523 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.613274 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 16:47:58.689017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.688967 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" event={"ID":"0641c0c8-4413-4dbc-a46e-8612fda4f05b","Type":"ContainerStarted","Data":"d63be1b7ebe6c4a7662b6723ad42a55bfa01862405e924fc8f4713bc7fd41a44"} Nov 03 16:47:58.691067 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.690993 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:47:58.740328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.740296 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:47:58.757164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:58.757110 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" podStartSLOduration=2.340230617 podStartE2EDuration="2.75709444s" totalImagesPullingTime="416.863823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:58.755987527 +0000 UTC m=+2022.449258587" watchObservedRunningTime="2026-11-03 16:47:58.75709444 +0000 UTC m=+2022.450365464" Nov 03 16:47:59.611849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:59.611813 4539 patch_prober.go:29] interesting pod/e2e-api-b-87b132-openshift-default-546b679549-rmj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 16:47:59.612155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:47:59.612123 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 16:48:00.172054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:00.172004 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:00.333946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:00.333870 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:00.475818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:00.475730 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:00.608907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:00.608864 4539 patch_prober.go:29] interesting pod/e2e-api-b-87b132-openshift-default-546b679549-rmj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 03 16:48:00.611599 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:00.611554 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 03 16:48:00.861008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:00.860978 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:01.606634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:01.606602 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:48:01.607886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:01.607866 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:48:02.300450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:02.300401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:04.316313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:04.316282 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:05.557071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:05.557038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:05.718545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:05.718502 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:05.766635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:05.766603 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:48:06.679579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:06.679496 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:06.831059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:06.831020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:07.521901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:07.521868 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:08.545785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:08.545746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:10.702790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:10.702753 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:11.420777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:11.420738 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:12.382938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:12.382900 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:13.711093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:13.711055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:13.879195 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:13.879111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:14.027782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:14.027746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:14.608526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:14.608484 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:14.756297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:14.756266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:14.907930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:14.907847 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:16.487364 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:16.487328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:18.377050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:18.377015 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:19.693715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:19.693674 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7"] Nov 03 16:48:19.694240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:19.693913 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" podUID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" containerName="maas-api" containerID="cri-o://6fb29d40fd39e588eef2c541fb80ffc380342269d712c4d32563a3b36c821de0" gracePeriod=30 Nov 03 16:48:19.812633 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:19.812578 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:19.921469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:19.921168 4539 generic.go:356] "Generic (PLEG): container finished" podID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" containerID="6fb29d40fd39e588eef2c541fb80ffc380342269d712c4d32563a3b36c821de0" exitCode=0 Nov 03 16:48:19.921469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:19.921363 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" event={"ID":"dd6f3cc8-fecb-4f11-9e06-5c3f24001984","Type":"ContainerDied","Data":"6fb29d40fd39e588eef2c541fb80ffc380342269d712c4d32563a3b36c821de0"} Nov 03 16:48:20.129822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.129791 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:48:20.188491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.188029 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs\") pod \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " Nov 03 16:48:20.188491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.188147 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca\") pod \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " Nov 03 16:48:20.188491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.188234 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls\") pod \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " Nov 03 16:48:20.188491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.188297 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4\" (UniqueName: \"kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4\") pod \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\" (UID: \"dd6f3cc8-fecb-4f11-9e06-5c3f24001984\") " Nov 03 16:48:20.191917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.191871 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca" pod "dd6f3cc8-fecb-4f11-9e06-5c3f24001984" (UID: "dd6f3cc8-fecb-4f11-9e06-5c3f24001984"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:20.192492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.192451 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4" pod "dd6f3cc8-fecb-4f11-9e06-5c3f24001984" (UID: "dd6f3cc8-fecb-4f11-9e06-5c3f24001984"). InnerVolumeSpecName "kube-api-access-dpxm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:20.212310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.211569 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs" pod "dd6f3cc8-fecb-4f11-9e06-5c3f24001984" (UID: "dd6f3cc8-fecb-4f11-9e06-5c3f24001984"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:20.212310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.211598 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls" pod "dd6f3cc8-fecb-4f11-9e06-5c3f24001984" (UID: "dd6f3cc8-fecb-4f11-9e06-5c3f24001984"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:20.325163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.324932 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:20.325163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.324967 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:20.325163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.324983 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:20.325163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.324999 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxm4\" (UniqueName: \"kubernetes.io/projected/dd6f3cc8-fecb-4f11-9e06-5c3f24001984-kube-api-access-dpxm4\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:20.636662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.636602 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:20.936092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.936052 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" event={"ID":"dd6f3cc8-fecb-4f11-9e06-5c3f24001984","Type":"ContainerDied","Data":"6b0bdf1dc559c442f115e05335990d31ce2548c6d1e6764831152ca334f78961"} Nov 03 16:48:20.936522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.936099 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" Nov 03 16:48:20.936522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.936125 4539 scope.go:122] "RemoveContainer" containerID="6fb29d40fd39e588eef2c541fb80ffc380342269d712c4d32563a3b36c821de0" Nov 03 16:48:20.983501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.983470 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7"] Nov 03 16:48:20.995202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:20.995157 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7"] Nov 03 16:48:22.429027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:22.428994 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:22.471333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:22.471300 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" path="/var/lib/kubelet/pods/dd6f3cc8-fecb-4f11-9e06-5c3f24001984/volumes" Nov 03 16:48:22.944839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:22.944801 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-87b132-86786c655c-z72n7" podUID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.105:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:48:23.131541 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:23.131509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:23.275603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:23.275567 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:24.023196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:24.023158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:24.188094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:24.188057 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:24.465144 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:24.465049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:24.628449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:24.626895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:24.948956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:24.948922 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:25.107275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.107235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:25.339475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.338344 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:25.348504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.348466 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf"] Nov 03 16:48:25.349778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.349754 4539 cpu_manager.go:414] "containerMap: removing container" podUID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" containerName="maas-api" Nov 03 16:48:25.349936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.349924 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" containerName="maas-api" Nov 03 16:48:25.350200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.350188 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6f3cc8-fecb-4f11-9e06-5c3f24001984" containerName="maas-api" Nov 03 16:48:25.356685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.356666 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.372105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.372077 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9160f5-openshift-default-dockercfg-vp7kc\"" Nov 03 16:48:25.389418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.389378 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf"] Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411247 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411362 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411405 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411483 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411550 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411659 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411708 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnddb\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411741 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.412152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.411789 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.458477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.458414 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q"] Nov 03 16:48:25.459249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.459219 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" containerID="cri-o://e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae" gracePeriod=30 Nov 03 16:48:25.487446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.486025 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.512835 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.512950 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513000 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnddb\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513028 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513070 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513162 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513213 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513248 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.513447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513312 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.514045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.513786 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.516330 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.515018 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.516330 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.515632 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.516330 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.516111 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.518533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.518502 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.529244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.528082 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.529244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.529066 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.544522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.543769 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.616509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.605147 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnddb\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb\") pod \"e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.640452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.637672 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:25.678492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.676016 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:25.802608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:25.796114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:26.070147 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.070107 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf"] Nov 03 16:48:26.071285 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:48:26.071250 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c00b5a6_8e87_44a1_bd0f_eea4b2dd7450.slice/crio-34a76c180e4e200c3cef4dfb553cf70923011be55186209febaa2b1892bacdb3 WatchSource:0}: Error finding container 34a76c180e4e200c3cef4dfb553cf70923011be55186209febaa2b1892bacdb3: Status 404 returned error can't find the container with id 34a76c180e4e200c3cef4dfb553cf70923011be55186209febaa2b1892bacdb3 Nov 03 16:48:26.073904 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.073873 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:48:26.073988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.073942 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:48:26.073988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.073969 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:48:26.164663 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:26.164623 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:48:26.526518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.526483 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:26.733466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.732839 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:26.989352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.989247 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" event={"ID":"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450","Type":"ContainerStarted","Data":"962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea"} Nov 03 16:48:26.989352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:26.989317 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" event={"ID":"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450","Type":"ContainerStarted","Data":"34a76c180e4e200c3cef4dfb553cf70923011be55186209febaa2b1892bacdb3"} Nov 03 16:48:27.030327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.030287 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:27.058780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.058718 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podStartSLOduration=2.058698747 podStartE2EDuration="2.058698747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:27.044535713 +0000 UTC m=+2050.737806747" watchObservedRunningTime="2026-11-03 16:48:27.058698747 +0000 UTC m=+2050.751969769" Nov 03 16:48:27.188899 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.188867 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:27.676891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.676853 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:27.683210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.683185 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:27.997195 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.997044 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:27.998077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:27.998052 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:48:29.013029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:29.012943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:29.599321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:29.599289 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:29.796356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:29.796326 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:30.251181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.251153 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:30.456129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.456098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:30.869865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.869780 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:30.895847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.895645 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:48:30.994213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994185 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.994479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994238 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.994479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994271 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.994479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994294 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.994479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994410 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.995005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994498 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.995005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994555 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.995005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994630 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:30.995005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994644 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.995005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994724 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo\") pod \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\" (UID: \"d6965863-e14b-43ed-afcb-f1b8c24caaa9\") " Nov 03 16:48:30.995005 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.994841 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:30.995320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995051 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:30.995320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995198 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:30.995439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995342 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:30.995558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995538 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:30.995621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995563 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:30.995621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995579 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:30.995621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995592 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:30.995621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:30.995605 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:31.001701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.001662 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:48:31.003565 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.003530 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "kube-api-access-2mzj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:31.003779 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.003750 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:31.004158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.004135 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token" pod "d6965863-e14b-43ed-afcb-f1b8c24caaa9" (UID: "d6965863-e14b-43ed-afcb-f1b8c24caaa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:31.028716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.028677 4539 generic.go:356] "Generic (PLEG): container finished" podID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerID="e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae" exitCode=0 Nov 03 16:48:31.028936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.028715 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" event={"ID":"d6965863-e14b-43ed-afcb-f1b8c24caaa9","Type":"ContainerDied","Data":"e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae"} Nov 03 16:48:31.028936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.028740 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" Nov 03 16:48:31.028936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.028779 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q" event={"ID":"d6965863-e14b-43ed-afcb-f1b8c24caaa9","Type":"ContainerDied","Data":"155b7ceffa8e0bce9361d90b2e26bbe72077a19fa363512e58a9a60f09314324"} Nov 03 16:48:31.028936 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.028806 4539 scope.go:122] "RemoveContainer" containerID="e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae" Nov 03 16:48:31.042333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.042311 4539 scope.go:122] "RemoveContainer" containerID="e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640" Nov 03 16:48:31.053035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.053012 4539 scope.go:122] "RemoveContainer" containerID="e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae" Nov 03 16:48:31.053409 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:31.053387 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae\": container with ID starting with e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae not found: ID does not exist" containerID="e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae" Nov 03 16:48:31.053505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.053435 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae"} err="failed to get container status \"e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae\": rpc error: code = NotFound desc = could not find container \"e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae\": container with ID starting with e120ed6e47a483cc0413a72aa4ad8664a4757f9df8b050810e93362fc3523aae not found: ID does not exist" Nov 03 16:48:31.053505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.053457 4539 scope.go:122] "RemoveContainer" containerID="e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640" Nov 03 16:48:31.053743 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:31.053722 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640\": container with ID starting with e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640 not found: ID does not exist" containerID="e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640" Nov 03 16:48:31.053806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.053755 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640"} err="failed to get container status \"e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640\": rpc error: code = NotFound desc = could not find container \"e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640\": container with ID starting with e1213ea7e66dc6f2fd5a32bd5a4f554d9c0dc299d9141f04697483486509f640 not found: ID does not exist" Nov 03 16:48:31.088157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.088120 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q"] Nov 03 16:48:31.091710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.091682 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-87b132-openshift-default-546b679549-rmj5q"] Nov 03 16:48:31.097123 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.097102 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:31.097123 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.097123 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:31.097287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.097134 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzj7\" (UniqueName: \"kubernetes.io/projected/d6965863-e14b-43ed-afcb-f1b8c24caaa9-kube-api-access-2mzj7\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:31.097287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.097143 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6965863-e14b-43ed-afcb-f1b8c24caaa9-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:31.542773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.542739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:31.735887 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.735840 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:31.903450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:31.896677 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:32.469301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:32.469272 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" path="/var/lib/kubelet/pods/d6965863-e14b-43ed-afcb-f1b8c24caaa9/volumes" Nov 03 16:48:33.110363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:33.110324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:33.795234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:33.795202 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p"] Nov 03 16:48:33.795531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:33.795500 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" podUID="9c50412e-f78a-40da-87f2-ac72644f1f17" containerName="maas-api" containerID="cri-o://c9ab10e4a682d8e50ec3fd3b5665ea5ef2af7d82d564975df204bb1e16a01449" gracePeriod=30 Nov 03 16:48:33.884807 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:33.884730 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:34.037361 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.037278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:34.065528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.062192 4539 generic.go:356] "Generic (PLEG): container finished" podID="9c50412e-f78a-40da-87f2-ac72644f1f17" containerID="c9ab10e4a682d8e50ec3fd3b5665ea5ef2af7d82d564975df204bb1e16a01449" exitCode=0 Nov 03 16:48:34.065528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.062248 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" event={"ID":"9c50412e-f78a-40da-87f2-ac72644f1f17","Type":"ContainerDied","Data":"c9ab10e4a682d8e50ec3fd3b5665ea5ef2af7d82d564975df204bb1e16a01449"} Nov 03 16:48:34.190282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.190251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:34.204068 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.204042 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:48:34.266804 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.266769 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls\") pod \"9c50412e-f78a-40da-87f2-ac72644f1f17\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " Nov 03 16:48:34.266998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.266888 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk\" (UniqueName: \"kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk\") pod \"9c50412e-f78a-40da-87f2-ac72644f1f17\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " Nov 03 16:48:34.266998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.266967 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca\") pod \"9c50412e-f78a-40da-87f2-ac72644f1f17\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " Nov 03 16:48:34.267140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.267074 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs\") pod \"9c50412e-f78a-40da-87f2-ac72644f1f17\" (UID: \"9c50412e-f78a-40da-87f2-ac72644f1f17\") " Nov 03 16:48:34.267503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.267466 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca" pod "9c50412e-f78a-40da-87f2-ac72644f1f17" (UID: "9c50412e-f78a-40da-87f2-ac72644f1f17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:34.267803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.267777 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c50412e-f78a-40da-87f2-ac72644f1f17-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.269270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.269237 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls" pod "9c50412e-f78a-40da-87f2-ac72644f1f17" (UID: "9c50412e-f78a-40da-87f2-ac72644f1f17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:34.270358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.270334 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs" pod "9c50412e-f78a-40da-87f2-ac72644f1f17" (UID: "9c50412e-f78a-40da-87f2-ac72644f1f17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:34.270469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.270359 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk" pod "9c50412e-f78a-40da-87f2-ac72644f1f17" (UID: "9c50412e-f78a-40da-87f2-ac72644f1f17"). InnerVolumeSpecName "kube-api-access-kfhwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:34.368697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.368620 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.368697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.368650 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhwk\" (UniqueName: \"kubernetes.io/projected/9c50412e-f78a-40da-87f2-ac72644f1f17-kube-api-access-kfhwk\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.368697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.368659 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c50412e-f78a-40da-87f2-ac72644f1f17-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.643477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.643320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:34.823636 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:34.823602 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:35.069800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:35.069770 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" Nov 03 16:48:35.070012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:35.069817 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p" event={"ID":"9c50412e-f78a-40da-87f2-ac72644f1f17","Type":"ContainerDied","Data":"9f760390108b7260784f808b1519c03065a8a861a7e44a12b98373319a3bda4e"} Nov 03 16:48:35.070012 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:35.069881 4539 scope.go:122] "RemoveContainer" containerID="c9ab10e4a682d8e50ec3fd3b5665ea5ef2af7d82d564975df204bb1e16a01449" Nov 03 16:48:35.120073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:35.120042 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p"] Nov 03 16:48:35.129640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:35.129612 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0a112-5c784b9dcc-4hc5p"] Nov 03 16:48:36.475451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:36.474638 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c50412e-f78a-40da-87f2-ac72644f1f17" path="/var/lib/kubelet/pods/9c50412e-f78a-40da-87f2-ac72644f1f17/volumes" Nov 03 16:48:36.550980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:36.550934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:38.270128 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:38.270087 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:39.966108 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:39.966070 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:40.536082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:40.536045 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:40.687537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:40.687499 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:41.231734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:41.231702 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:41.511863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:41.511749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:41.557527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:41.557489 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn"] Nov 03 16:48:41.557786 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:41.557760 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" containerID="cri-o://5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb" gracePeriod=30 Nov 03 16:48:41.655920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:41.655891 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:42.147006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:42.146977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:43.247155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.247120 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.652273 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6"] Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653078 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653096 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653129 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9c50412e-f78a-40da-87f2-ac72644f1f17" containerName="maas-api" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653137 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c50412e-f78a-40da-87f2-ac72644f1f17" containerName="maas-api" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653151 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653159 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653288 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653304 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c50412e-f78a-40da-87f2-ac72644f1f17" containerName="maas-api" Nov 03 16:48:43.655185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.653313 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6965863-e14b-43ed-afcb-f1b8c24caaa9" containerName="istio-proxy" Nov 03 16:48:43.657979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.657957 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.683489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.679493 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d6bb42-openshift-default-dockercfg-fffgm\"" Nov 03 16:48:43.702833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.702795 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6"] Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.743832 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.743963 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.743986 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.744125 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.744167 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.744202 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvzkk\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.744253 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.744300 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.746470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.744328 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.778178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.775837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852493 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852565 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852610 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzkk\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852640 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852732 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852789 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.852931 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.853079 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.853109 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.854207 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.854536 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.854931 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.855821 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.856519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.856089 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.880738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.866630 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.885327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.885259 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzkk\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.885914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.885888 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:43.894450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:43.891014 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token\") pod \"e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:44.007470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.006677 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:44.130152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.129798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:44.357792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.357758 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6"] Nov 03 16:48:44.358254 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:48:44.357786 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6167316_2853_40c1_9357_1b10c4b471a5.slice/crio-42ba454dbcc072df5e79bd36f37826f5fe62ea4482a62ae6c55c3ab42513b946 WatchSource:0}: Error finding container 42ba454dbcc072df5e79bd36f37826f5fe62ea4482a62ae6c55c3ab42513b946: Status 404 returned error can't find the container with id 42ba454dbcc072df5e79bd36f37826f5fe62ea4482a62ae6c55c3ab42513b946 Nov 03 16:48:44.361047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.361007 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:48:44.361136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.361117 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:48:44.361176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.361159 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:48:44.528404 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:44.528313 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:45.160593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:45.160555 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" event={"ID":"f6167316-2853-40c1-9357-1b10c4b471a5","Type":"ContainerStarted","Data":"b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370"} Nov 03 16:48:45.160821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:45.160625 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" event={"ID":"f6167316-2853-40c1-9357-1b10c4b471a5","Type":"ContainerStarted","Data":"42ba454dbcc072df5e79bd36f37826f5fe62ea4482a62ae6c55c3ab42513b946"} Nov 03 16:48:45.206435 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:45.206367 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podStartSLOduration=2.206348088 podStartE2EDuration="2.206348088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:45.204883505 +0000 UTC m=+2068.898154528" watchObservedRunningTime="2026-11-03 16:48:45.206348088 +0000 UTC m=+2068.899619111" Nov 03 16:48:46.007672 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.007637 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:46.013530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.013506 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:46.167314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.167279 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:46.168176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.168154 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:48:46.338922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.338877 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:46.845869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.845835 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:48:46.914790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.914754 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.914989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.914817 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.914989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.914895 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.914989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.914928 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.914989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.914981 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.915188 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915035 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.915237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915211 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.915344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915326 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.915415 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915400 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy\") pod \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\" (UID: \"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8\") " Nov 03 16:48:46.915494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915466 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:46.915740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915717 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:46.915912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915892 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:46.916008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.915989 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:46.916059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.916017 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:46.916059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.916035 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:46.916157 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.916067 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:46.916412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.916382 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:46.918845 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.918819 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:46.919593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.919556 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:46.919714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.919648 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:48:46.930992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:46.930904 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p" pod "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" (UID: "49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8"). InnerVolumeSpecName "kube-api-access-vt75p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:47.017417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.017375 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.017417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.017411 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt75p\" (UniqueName: \"kubernetes.io/projected/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-kube-api-access-vt75p\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.017417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.017460 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.018069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.017472 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.018069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.017481 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.018069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.017490 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.175013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.174981 4539 generic.go:356] "Generic (PLEG): container finished" podID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerID="5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb" exitCode=0 Nov 03 16:48:47.175217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.175030 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" Nov 03 16:48:47.175217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.175032 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" event={"ID":"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8","Type":"ContainerDied","Data":"5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb"} Nov 03 16:48:47.175217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.175086 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn" event={"ID":"49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8","Type":"ContainerDied","Data":"df6397e822ad91c500d67f05532e4e967cffc254e46f63dffc437c49938ee1aa"} Nov 03 16:48:47.175217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.175110 4539 scope.go:122] "RemoveContainer" containerID="5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb" Nov 03 16:48:47.186292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.186274 4539 scope.go:122] "RemoveContainer" containerID="9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f" Nov 03 16:48:47.196646 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.196628 4539 scope.go:122] "RemoveContainer" containerID="5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb" Nov 03 16:48:47.197002 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:47.196978 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb\": container with ID starting with 5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb not found: ID does not exist" containerID="5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb" Nov 03 16:48:47.197088 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.197015 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb"} err="failed to get container status \"5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb\": rpc error: code = NotFound desc = could not find container \"5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb\": container with ID starting with 5b497d85aa86ca76dfc9ed5e93f221723f0ad33984255ba6141709aaa416e5fb not found: ID does not exist" Nov 03 16:48:47.197088 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.197038 4539 scope.go:122] "RemoveContainer" containerID="9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f" Nov 03 16:48:47.197317 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:47.197296 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f\": container with ID starting with 9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f not found: ID does not exist" containerID="9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f" Nov 03 16:48:47.197371 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.197327 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f"} err="failed to get container status \"9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f\": rpc error: code = NotFound desc = could not find container \"9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f\": container with ID starting with 9b4f7035e8d0e37759e4f8c63d8521c65b93458080ef998f1a8cbe71a880078f not found: ID does not exist" Nov 03 16:48:47.267489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.267394 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn"] Nov 03 16:48:47.277718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.277686 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e0a112-openshift-default-868b8956c7-h9lwn"] Nov 03 16:48:47.711273 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.711234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:47.869111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:47.869069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:48.040105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:48.037524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:48.209911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:48.209881 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:48.472333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:48.472253 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" path="/var/lib/kubelet/pods/49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8/volumes" Nov 03 16:48:49.058577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:49.058542 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:49.517766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:49.517683 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:49.681214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:49.681179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:51.485790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:51.485757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:53.344107 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344072 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l"] Nov 03 16:48:53.344698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344683 4539 cpu_manager.go:414] "containerMap: removing container" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" Nov 03 16:48:53.344749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344700 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" Nov 03 16:48:53.344749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344715 4539 cpu_manager.go:414] "containerMap: removing container" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" Nov 03 16:48:53.344749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344721 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" Nov 03 16:48:53.344849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344814 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" Nov 03 16:48:53.344849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.344828 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a9fad8-f0eb-405e-a3c8-25ca54b6a9e8" containerName="istio-proxy" Nov 03 16:48:53.349876 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.349856 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.364868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.364847 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9160f5-serving-cert\"" Nov 03 16:48:53.382402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.382367 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l"] Nov 03 16:48:53.427644 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.427608 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5h6\" (UniqueName: \"kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.428003 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.427982 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.428481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.428232 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.428596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.428546 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.461256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.461219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:53.530976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.530081 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.530976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.530321 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5h6\" (UniqueName: \"kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.530976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.530448 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.530976 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.530605 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.531740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.531716 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.547094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.547058 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.547438 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.547397 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.556263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.556230 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5h6\" (UniqueName: \"kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6\") pod \"maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.608056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.607930 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:53.665669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.665637 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:53.750035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.749971 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:53.974867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:53.974839 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l"] Nov 03 16:48:53.976088 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:48:53.976065 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825917b2_9640_4c6a_bd63_d1adbe2e7631.slice/crio-e8fab9733a8be32b774acca8ae09d7db3ae8a30326836eca8ca9e999b1d031f4 WatchSource:0}: Error finding container e8fab9733a8be32b774acca8ae09d7db3ae8a30326836eca8ca9e999b1d031f4: Status 404 returned error can't find the container with id e8fab9733a8be32b774acca8ae09d7db3ae8a30326836eca8ca9e999b1d031f4 Nov 03 16:48:54.233490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:54.233367 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" event={"ID":"825917b2-9640-4c6a-bd63-d1adbe2e7631","Type":"ContainerStarted","Data":"e8fab9733a8be32b774acca8ae09d7db3ae8a30326836eca8ca9e999b1d031f4"} Nov 03 16:48:55.120485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:55.113156 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:55.242808 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:55.242770 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" event={"ID":"825917b2-9640-4c6a-bd63-d1adbe2e7631","Type":"ContainerStarted","Data":"966796f5259ca16206af42011785039f4f9d2a9baa1398f8a5a8784e3224766e"} Nov 03 16:48:55.243017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:55.242872 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:48:55.282710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:55.282651 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" podStartSLOduration=1.936863989 podStartE2EDuration="2.282634587s" totalImagesPullingTime="345.770598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:55.281856756 +0000 UTC m=+2078.975127780" watchObservedRunningTime="2026-11-03 16:48:55.282634587 +0000 UTC m=+2078.975905609" Nov 03 16:48:56.158466 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:56.158393 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:48:57.085585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.085529 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp"] Nov 03 16:48:57.090044 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.090019 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.124857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.124814 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp"] Nov 03 16:48:57.125046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.124929 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5a7d95-serving-cert\"" Nov 03 16:48:57.183650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.183615 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.184027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.183732 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.184027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.183819 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.184027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.183916 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pb5\" (UniqueName: \"kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.209086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.209055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.285297 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.285419 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.285530 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pb5\" (UniqueName: \"kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.285659 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:57.285809 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-serving-cert: secret "maas-api-e2e-ipp-a-5a7d95-serving-cert" not found Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:48:57.285880 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls podName:eea3c944-079d-4f52-955b-bbd2f4d9e311 nodeName:}" failed. No retries permitted until 2026-11-03 16:48:57.785861423 +0000 UTC m=+2081.479132424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls") pod "maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" (UID: "eea3c944-079d-4f52-955b-bbd2f4d9e311") : secret "maas-api-e2e-ipp-a-5a7d95-serving-cert" not found Nov 03 16:48:57.286341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.286215 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.289384 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.289361 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.313583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.313548 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pb5\" (UniqueName: \"kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.349635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.349551 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:57.537871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.537831 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:57.792483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.792437 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:57.795367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:57.795333 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:58.004938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:58.004905 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:58.329183 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:58.329156 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp"] Nov 03 16:48:58.330192 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:48:58.330166 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea3c944_079d_4f52_955b_bbd2f4d9e311.slice/crio-bafcf89fa658143657266509e3773ebd471b19a4dda84f37f5bbac659ce5770c WatchSource:0}: Error finding container bafcf89fa658143657266509e3773ebd471b19a4dda84f37f5bbac659ce5770c: Status 404 returned error can't find the container with id bafcf89fa658143657266509e3773ebd471b19a4dda84f37f5bbac659ce5770c Nov 03 16:48:58.948016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:58.947983 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:48:59.273816 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:59.273784 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" event={"ID":"eea3c944-079d-4f52-955b-bbd2f4d9e311","Type":"ContainerStarted","Data":"0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098"} Nov 03 16:48:59.274029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:59.273834 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" event={"ID":"eea3c944-079d-4f52-955b-bbd2f4d9e311","Type":"ContainerStarted","Data":"bafcf89fa658143657266509e3773ebd471b19a4dda84f37f5bbac659ce5770c"} Nov 03 16:48:59.274029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:59.273977 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:48:59.319537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:48:59.319487 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" podStartSLOduration=1.935568039 podStartE2EDuration="2.319472512s" totalImagesPullingTime="383.904473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:59.317579328 +0000 UTC m=+2083.010850349" watchObservedRunningTime="2026-11-03 16:48:59.319472512 +0000 UTC m=+2083.012743546" Nov 03 16:49:00.494092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:00.494060 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:01.256965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:01.256938 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:49:02.085267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:02.085236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:03.713856 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:03.713819 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:04.887181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:04.887148 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:05.030746 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:05.030713 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:05.324737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:05.324709 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:49:06.024300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:06.024268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:06.476835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:06.476754 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:07.599235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:07.599200 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:09.179969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:09.179932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:10.805405 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:10.805373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:11.513234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:11.513201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:11.697526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:11.697496 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2"] Nov 03 16:49:11.697794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:11.697770 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" containerID="cri-o://dae5b5e676ee7b6b3785457e1529d1e00fb84aa4fe85be07ede488404f796730" gracePeriod=30 Nov 03 16:49:12.842641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:12.842600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:12.997241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:12.997132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:13.151341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:13.151308 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:13.300970 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:13.300933 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:14.881509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:14.881477 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:16.427728 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.427697 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq"] Nov 03 16:49:16.428536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.428505 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" podUID="f5fb30bb-b719-4f38-8649-7d51248da1bb" containerName="maas-api" containerID="cri-o://881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1" gracePeriod=30 Nov 03 16:49:16.517457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.517404 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:16.748004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.747970 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:16.819136 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.819081 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:49:16.820790 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.820722 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:49:16.846342 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.846312 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:49:16.885442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.885389 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh\" (UniqueName: \"kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh\") pod \"f5fb30bb-b719-4f38-8649-7d51248da1bb\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " Nov 03 16:49:16.885637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.885516 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca\") pod \"f5fb30bb-b719-4f38-8649-7d51248da1bb\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " Nov 03 16:49:16.885710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.885645 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs\") pod \"f5fb30bb-b719-4f38-8649-7d51248da1bb\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " Nov 03 16:49:16.885767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.885708 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls\") pod \"f5fb30bb-b719-4f38-8649-7d51248da1bb\" (UID: \"f5fb30bb-b719-4f38-8649-7d51248da1bb\") " Nov 03 16:49:16.885937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.885909 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca" pod "f5fb30bb-b719-4f38-8649-7d51248da1bb" (UID: "f5fb30bb-b719-4f38-8649-7d51248da1bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:16.886224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.886201 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb30bb-b719-4f38-8649-7d51248da1bb-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:16.897596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.888146 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls" pod "f5fb30bb-b719-4f38-8649-7d51248da1bb" (UID: "f5fb30bb-b719-4f38-8649-7d51248da1bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:16.897596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.888608 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh" pod "f5fb30bb-b719-4f38-8649-7d51248da1bb" (UID: "f5fb30bb-b719-4f38-8649-7d51248da1bb"). InnerVolumeSpecName "kube-api-access-4lmnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:16.897596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.889039 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs" pod "f5fb30bb-b719-4f38-8649-7d51248da1bb" (UID: "f5fb30bb-b719-4f38-8649-7d51248da1bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:16.909230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.905740 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:16.987482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.987405 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:16.987482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.987449 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fb30bb-b719-4f38-8649-7d51248da1bb-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:16.987482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.987459 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmnh\" (UniqueName: \"kubernetes.io/projected/f5fb30bb-b719-4f38-8649-7d51248da1bb-kube-api-access-4lmnh\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:16.988040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:16.988024 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:49:17.088833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.088794 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.088875 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.088913 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.088980 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.089038 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.089183 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.089212 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.089246 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.089338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.089313 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket\") pod \"40e8a12c-f0a1-4690-ab57-56958719f4ba\" (UID: \"40e8a12c-f0a1-4690-ab57-56958719f4ba\") " Nov 03 16:49:17.090155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.090126 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:17.090346 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.090328 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:17.090510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.090489 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:17.091231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.090956 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:17.091231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.091205 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:17.091577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.091394 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.091577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.091443 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.091577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.091457 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8a12c-f0a1-4690-ab57-56958719f4ba-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.091577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.091471 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.091577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.091486 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.093969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.093929 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "kube-api-access-7w9ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:17.094888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.094159 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:17.094888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.094274 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:17.097588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.097562 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy" pod "40e8a12c-f0a1-4690-ab57-56958719f4ba" (UID: "40e8a12c-f0a1-4690-ab57-56958719f4ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:17.168553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.168508 4539 scope.go:122] "RemoveContainer" containerID="72ff845877700413c53a4664fc691d7d224f85211bebb9adc16b42ad9dc505f2" Nov 03 16:49:17.184539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.184511 4539 scope.go:122] "RemoveContainer" containerID="dae5b5e676ee7b6b3785457e1529d1e00fb84aa4fe85be07ede488404f796730" Nov 03 16:49:17.193314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.193287 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.193536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.193517 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w9ww\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-kube-api-access-7w9ww\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.193680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.193653 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.193769 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.193682 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8a12c-f0a1-4690-ab57-56958719f4ba-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.406251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406219 4539 generic.go:356] "Generic (PLEG): container finished" podID="f5fb30bb-b719-4f38-8649-7d51248da1bb" containerID="881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1" exitCode=0 Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406309 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406311 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" event={"ID":"f5fb30bb-b719-4f38-8649-7d51248da1bb","Type":"ContainerDied","Data":"881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1"} Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406376 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq" event={"ID":"f5fb30bb-b719-4f38-8649-7d51248da1bb","Type":"ContainerDied","Data":"14150150586c5e274e2de23787cba9a761d5b6c9bd616bc9dc4e29b020025d8b"} Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406399 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" event={"ID":"40e8a12c-f0a1-4690-ab57-56958719f4ba","Type":"ContainerDied","Data":"dae5b5e676ee7b6b3785457e1529d1e00fb84aa4fe85be07ede488404f796730"} Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406457 4539 scope.go:122] "RemoveContainer" containerID="881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1" Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406462 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" event={"ID":"40e8a12c-f0a1-4690-ab57-56958719f4ba","Type":"ContainerDied","Data":"530c96e1519949c5aeb46dd7a1db27fdd7fb40bc1e08c319346cefcf56ef3aab"} Nov 03 16:49:17.406502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.406486 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2" Nov 03 16:49:17.432653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.432617 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:17.437085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.436955 4539 scope.go:122] "RemoveContainer" containerID="881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1" Nov 03 16:49:17.437482 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:49:17.437417 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1\": container with ID starting with 881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1 not found: ID does not exist" containerID="881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1" Nov 03 16:49:17.437579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.437501 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1"} err="failed to get container status \"881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1\": rpc error: code = NotFound desc = could not find container \"881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1\": container with ID starting with 881a909ae7ac616f9a0fb1d82cad617cf03099fe8a1e8ed787ae23e617f140c1 not found: ID does not exist" Nov 03 16:49:17.507666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.507633 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2"] Nov 03 16:49:17.511733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.511706 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-g2ss2"] Nov 03 16:49:17.542126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.542098 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq"] Nov 03 16:49:17.552149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:17.552107 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-624649-869b67df6c-hbdqq"] Nov 03 16:49:18.469033 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:18.468995 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" path="/var/lib/kubelet/pods/40e8a12c-f0a1-4690-ab57-56958719f4ba/volumes" Nov 03 16:49:18.469506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:18.469491 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fb30bb-b719-4f38-8649-7d51248da1bb" path="/var/lib/kubelet/pods/f5fb30bb-b719-4f38-8649-7d51248da1bb/volumes" Nov 03 16:49:19.564861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:19.564829 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:19.715447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:19.714886 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:20.369448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:20.367088 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:20.746957 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:20.746867 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:21.488021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.487984 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4"] Nov 03 16:49:21.488881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.488854 4539 cpu_manager.go:414] "containerMap: removing container" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" Nov 03 16:49:21.488881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.488881 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" Nov 03 16:49:21.489081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.488934 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f5fb30bb-b719-4f38-8649-7d51248da1bb" containerName="maas-api" Nov 03 16:49:21.489081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.488943 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fb30bb-b719-4f38-8649-7d51248da1bb" containerName="maas-api" Nov 03 16:49:21.489207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.489085 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" Nov 03 16:49:21.489207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.489108 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fb30bb-b719-4f38-8649-7d51248da1bb" containerName="maas-api" Nov 03 16:49:21.568450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.551532 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4"] Nov 03 16:49:21.568450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.551638 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.568450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.558230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:21.589529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.589499 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9947ab\"/\"openshift-service-ca.crt\"" Nov 03 16:49:21.589701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.589548 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9947ab\"/\"ipp-route-9947ab-kserve-self-signed-certs\"" Nov 03 16:49:21.589701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.589685 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9947ab\"/\"default-dockercfg-vcmzx\"" Nov 03 16:49:21.589844 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.589713 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9947ab\"/\"kube-root-ca.crt\"" Nov 03 16:49:21.695277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.695244 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.695536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.695301 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.695536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.695458 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.695536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.695509 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.695727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.695572 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.695727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.695638 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796lf\" (UniqueName: \"kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.711922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.711889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.800464 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.800544 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.800573 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.800617 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.800674 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.800707 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796lf\" (UniqueName: \"kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.801957 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.802447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.802310 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.803638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.802714 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.845285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.837620 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.845819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.845788 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.846711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.846686 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796lf\" (UniqueName: \"kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf\") pod \"ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:21.869827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:21.869794 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:22.297562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:22.297527 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4"] Nov 03 16:49:22.435115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:22.434866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:22.493486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:22.493231 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" event={"ID":"1a3e0dcd-871f-4076-bd65-4c6f99d95201","Type":"ContainerStarted","Data":"8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696"} Nov 03 16:49:22.493486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:22.493277 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" event={"ID":"1a3e0dcd-871f-4076-bd65-4c6f99d95201","Type":"ContainerStarted","Data":"d0342df1cf8c6c6790695359a76e5b20288a38baa57e88ff07cd05e4843e53a5"} Nov 03 16:49:22.534118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:22.529912 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" podStartSLOduration=1.5298904119999999 podStartE2EDuration="1.529890412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:22.524953503 +0000 UTC m=+2106.218224522" watchObservedRunningTime="2026-11-03 16:49:22.529890412 +0000 UTC m=+2106.223161434" Nov 03 16:49:22.871117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:22.871064 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:24.186838 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:24.186784 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:26.085975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:26.085940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:26.183108 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:49:26.183076 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:49:27.530028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.529975 4539 generic.go:356] "Generic (PLEG): container finished" podID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerID="962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea" exitCode=0 Nov 03 16:49:27.530534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.530034 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" event={"ID":"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450","Type":"ContainerDied","Data":"962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea"} Nov 03 16:49:27.530721 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.530703 4539 scope.go:122] "RemoveContainer" containerID="962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea" Nov 03 16:49:27.532801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.532766 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:49:27.532904 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.532858 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:49:27.532942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.532907 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:49:27.750448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.749848 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:27.952198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.952103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:27.997287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:27.997253 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:49:28.371820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:28.371766 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:28.539392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:28.539315 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" event={"ID":"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450","Type":"ContainerStarted","Data":"23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34"} Nov 03 16:49:28.676839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:28.676746 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:49:28.677905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:28.677875 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 16:49:28.678057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:28.677936 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 16:49:29.677648 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:29.677610 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 16:49:29.678061 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:29.677680 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 16:49:29.681788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:29.681751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:30.677392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:30.677352 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 16:49:30.677593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:30.677414 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 16:49:31.677150 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:31.677114 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 16:49:31.677610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:31.677179 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 16:49:31.870944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:31.870905 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:31.898484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:31.895858 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:31.939711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:31.939583 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:32.589363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:32.589335 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:49:32.677589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:32.677558 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 16:49:32.678081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:32.677640 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 16:49:33.565022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:33.564993 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:33.677127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:33.677097 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 16:49:33.677350 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:33.677165 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 16:49:34.682150 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:34.682120 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:49:34.682652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:34.682616 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:49:34.683482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:34.683462 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:49:35.365745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:35.365710 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:36.973992 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:36.973945 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.156376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:37.156337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.306066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:37.306027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:37.781255 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:37.781193 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:39.536276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:39.536227 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:41.357732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:41.357685 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:41.523459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:41.523365 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:42.219999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:42.219950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:42.738086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:42.737989 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:43.225827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:43.225742 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:44.237640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:44.237605 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:44.752446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:44.752317 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:45.053772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:45.053710 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:45.232282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:45.232252 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:45.432727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:45.432638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:45.604886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:45.604836 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:45.834388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:45.834350 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:46.742304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:46.742253 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:46.892199 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:46.892103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:47.194016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.193514 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:49:47.194016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.193581 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:49:47.573495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.573397 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6"] Nov 03 16:49:47.574373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.574346 4539 cpu_manager.go:414] "containerMap: removing container" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" Nov 03 16:49:47.574373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.574371 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" Nov 03 16:49:47.574615 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.574530 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e8a12c-f0a1-4690-ab57-56958719f4ba" containerName="istio-proxy" Nov 03 16:49:47.580356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.578493 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.612711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.602196 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b9e362\"/\"kube-root-ca.crt\"" Nov 03 16:49:47.612711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.602182 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b9e362\"/\"praxis-route-b9e362-kserve-self-signed-certs\"" Nov 03 16:49:47.612711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.602320 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b9e362\"/\"openshift-service-ca.crt\"" Nov 03 16:49:47.612711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.602465 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b9e362\"/\"default-dockercfg-qn7m5\"" Nov 03 16:49:47.634068 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.634015 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6"] Nov 03 16:49:47.681674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.681642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:47.715176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.715130 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.715391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.715197 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.715391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.715348 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.715537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.715409 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.715587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.715554 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.715587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.715582 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmrv\" (UniqueName: \"kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817207 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817304 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817338 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmrv\" (UniqueName: \"kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817393 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817483 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817571 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.817823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.817783 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.818258 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.818235 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.818341 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.818286 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.820176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.820149 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.820690 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.820667 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.842596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.842495 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmrv\" (UniqueName: \"kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv\") pod \"praxis-route-b9e362-kserve-5dfc6c886c-t4km6\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:47.875593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.875548 4539 generic.go:356] "Generic (PLEG): container finished" podID="f6167316-2853-40c1-9357-1b10c4b471a5" containerID="b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370" exitCode=0 Nov 03 16:49:47.875798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.875623 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" event={"ID":"f6167316-2853-40c1-9357-1b10c4b471a5","Type":"ContainerDied","Data":"b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370"} Nov 03 16:49:47.876208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.876189 4539 scope.go:122] "RemoveContainer" containerID="b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370" Nov 03 16:49:47.878509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.878472 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:49:47.888754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.878571 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:49:47.888754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.878621 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:49:47.921729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:47.921453 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:48.169743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.168638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:48.429407 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.429362 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6"] Nov 03 16:49:48.460226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.460168 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:48.614085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.614053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:48.887639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.887602 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" event={"ID":"ba167e1b-d5ea-4874-a11c-6a09b710bd8b","Type":"ContainerStarted","Data":"9ab6d890d5f879682535c8121257cd4de5bcab5edb302330c7a928ceb1e56b85"} Nov 03 16:49:48.888239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.888215 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" event={"ID":"ba167e1b-d5ea-4874-a11c-6a09b710bd8b","Type":"ContainerStarted","Data":"9d8d2f15438a98ac83a5475380abc1c8c4ee84dfba700ea39bc79c6e75e67392"} Nov 03 16:49:48.891102 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.891069 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" event={"ID":"f6167316-2853-40c1-9357-1b10c4b471a5","Type":"ContainerStarted","Data":"2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a"} Nov 03 16:49:48.937500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.935188 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:48.952089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:48.952024 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" podStartSLOduration=1.9520009310000002 podStartE2EDuration="1.952000931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:48.948621562 +0000 UTC m=+2132.641892582" watchObservedRunningTime="2026-11-03 16:49:48.952000931 +0000 UTC m=+2132.645271952" Nov 03 16:49:49.014944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:49.014906 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:49:49.015340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:49.015316 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 16:49:49.015535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:49.015508 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 16:49:49.522913 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:49.522869 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:49.683409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:49.683359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:49.831731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:49.831675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:50.010858 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:50.010821 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 16:49:50.011536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:50.011504 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 16:49:50.354528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:50.354501 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:50.599223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:50.594183 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:50.990283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:50.990249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:51.021456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:51.009886 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 16:49:51.021456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:51.009951 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 16:49:52.030877 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:52.030836 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 16:49:52.031526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:52.031493 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 16:49:52.236450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:52.236010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:52.385890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:52.385780 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:53.009864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:53.009826 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 16:49:53.010150 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:53.010119 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 16:49:53.892868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:53.892818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:54.015451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:54.008355 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 16:49:54.015451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:54.008484 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 16:49:55.012610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:55.012576 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:49:55.013120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:55.013000 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:49:55.013953 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:55.013929 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:49:55.892043 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:55.891982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:56.158369 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:49:56.158278 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:49:57.494123 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:57.494079 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:49:57.926081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:57.926045 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:57.943622 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:57.943593 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:58.045115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:58.045089 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:49:59.082274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:49:59.082215 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:00.868829 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:00.868562 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:02.572998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:02.572965 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:04.262343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:04.262304 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:04.965609 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:04.965522 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:05.471848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:05.471820 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:05.651453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:05.647889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:06.041458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:06.040768 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:08.226836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:08.226798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:09.357821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:09.357781 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:09.514097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:09.514063 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:10.394855 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:10.394821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:10.557867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:10.557835 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:10.781589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:10.781560 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:10.931550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:10.931518 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:11.564237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:11.564206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:11.716532 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:11.716477 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:12.608531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:12.608495 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:14.116863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.116824 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr"] Nov 03 16:50:14.120926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.120901 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.134619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.134587 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-31de0f-serving-cert\"" Nov 03 16:50:14.150998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.150938 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr"] Nov 03 16:50:14.186322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.186278 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.186614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.186469 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.186699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.186657 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdhm\" (UniqueName: \"kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.186801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.186781 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.240677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.240642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:14.290220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.290177 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdhm\" (UniqueName: \"kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.290445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.290287 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.290445 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.290366 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.290576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.290483 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.290652 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:50:14.290634 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-serving-cert: secret "maas-api-e2e-praxis-b-31de0f-serving-cert" not found Nov 03 16:50:14.290736 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:50:14.290722 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls podName:7653bca9-7e28-4343-beff-c04c7e9b8be3 nodeName:}" failed. No retries permitted until 2026-11-03 16:50:14.790699711 +0000 UTC m=+2158.483970713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls") pod "maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" (UID: "7653bca9-7e28-4343-beff-c04c7e9b8be3") : secret "maas-api-e2e-praxis-b-31de0f-serving-cert" not found Nov 03 16:50:14.291954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.291919 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.328082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.328042 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.357804 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.354463 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdhm\" (UniqueName: \"kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.385826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.385736 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:14.798261 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.798225 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.800999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.800972 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:14.837068 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:14.837039 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:15.035861 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:15.035588 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:15.376404 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:15.376367 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr"] Nov 03 16:50:15.379034 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:50:15.378996 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7653bca9_7e28_4343_beff_c04c7e9b8be3.slice/crio-f3573ffa5beaee0630339c229acd6d84efe74067ba5fd8060906ba00d2365a17 WatchSource:0}: Error finding container f3573ffa5beaee0630339c229acd6d84efe74067ba5fd8060906ba00d2365a17: Status 404 returned error can't find the container with id f3573ffa5beaee0630339c229acd6d84efe74067ba5fd8060906ba00d2365a17 Nov 03 16:50:15.979552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:15.979452 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:16.194055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:16.194003 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" event={"ID":"7653bca9-7e28-4343-beff-c04c7e9b8be3","Type":"ContainerStarted","Data":"589c9fd3ad5daa722bf4969052d8fdd2443295fd02ccc11af260772be8ce7f8f"} Nov 03 16:50:16.194244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:16.194066 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" event={"ID":"7653bca9-7e28-4343-beff-c04c7e9b8be3","Type":"ContainerStarted","Data":"f3573ffa5beaee0630339c229acd6d84efe74067ba5fd8060906ba00d2365a17"} Nov 03 16:50:16.194244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:16.194215 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:16.241641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:16.241512 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" podStartSLOduration=1.899371371 podStartE2EDuration="2.241492029s" totalImagesPullingTime="342.120658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:16.241318592 +0000 UTC m=+2159.934589615" watchObservedRunningTime="2026-11-03 16:50:16.241492029 +0000 UTC m=+2159.934763052" Nov 03 16:50:17.608492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:17.608454 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:19.273878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:19.273850 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:21.334149 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:21.334106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:22.208417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:22.208382 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:50:22.969903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:22.969866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:24.642864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:24.642821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:26.158163 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:50:26.158118 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:50:26.195468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:26.195328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:27.805145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:27.805106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:28.631322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:28.631284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:28.780859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:28.780818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:29.382245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:29.382196 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:30.932083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:30.932041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:32.249508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:32.249476 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:33.006677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:33.006646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:33.150396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:33.150361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:33.303706 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:33.303670 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:34.761481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:34.761449 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:36.860750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:36.860716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:38.926825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:38.926791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:40.589043 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:40.589006 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:42.073747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:42.073706 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:42.298685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:42.298641 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:43.924169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:43.924130 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:45.573912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:45.573875 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:45.725164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:45.725129 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:46.043089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:46.043053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:46.220786 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:46.220751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:47.608143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:47.608076 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:47.867814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:47.867723 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:49.641537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:49.641499 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:49.827146 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:49.827105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:49.992765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:49.992682 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:51.134267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:51.134238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:51.390345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:51.390255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:51.584812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:51.584781 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:51.768064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:51.767974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:53.078035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:53.077992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:53.335375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:53.335269 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg"] Nov 03 16:50:53.335713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:53.335680 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" containerID="cri-o://c058f0dee5a1fd81c80154cb3eec67f9748a00b6a7fd6ee2bec29983dc7f5e5d" gracePeriod=30 Nov 03 16:50:54.007249 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:54.007214 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:54.160926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:54.160889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:54.308391 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:54.308357 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:54.842126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:54.839544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:56.278451 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:50:56.275076 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:50:56.701517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:56.701410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:57.988253 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:57.988218 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:50:57.988686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:57.988287 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:50:58.580590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.580509 4539 generic.go:356] "Generic (PLEG): container finished" podID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerID="c058f0dee5a1fd81c80154cb3eec67f9748a00b6a7fd6ee2bec29983dc7f5e5d" exitCode=0 Nov 03 16:50:58.580808 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.580634 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" event={"ID":"129681f8-da05-4f1f-978b-7998ab2c0c0a","Type":"ContainerDied","Data":"c058f0dee5a1fd81c80154cb3eec67f9748a00b6a7fd6ee2bec29983dc7f5e5d"} Nov 03 16:50:58.609219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.609021 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:50:58.674998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.674968 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:50:58.810493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810458 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.810824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810503 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.810824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810541 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.810824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810563 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.810824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810768 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.810824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810805 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.811110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810852 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:58.811110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810857 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:58.811110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810927 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.811110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.810975 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.811110 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811043 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket\") pod \"129681f8-da05-4f1f-978b-7998ab2c0c0a\" (UID: \"129681f8-da05-4f1f-978b-7998ab2c0c0a\") " Nov 03 16:50:58.811380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811172 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:58.811489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811464 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:58.811651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811618 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:58.811854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811830 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129681f8-da05-4f1f-978b-7998ab2c0c0a-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.811932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811895 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.812029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.811971 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.812164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.812041 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.812164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.812054 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.814918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.814881 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:50:58.814918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.814896 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "kube-api-access-lf8gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:58.815130 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.815049 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:50:58.816586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.816557 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token" pod "129681f8-da05-4f1f-978b-7998ab2c0c0a" (UID: "129681f8-da05-4f1f-978b-7998ab2c0c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:58.913937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.913827 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.913937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.913870 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.913937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.913884 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:58.913937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:58.913898 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf8gj\" (UniqueName: \"kubernetes.io/projected/129681f8-da05-4f1f-978b-7998ab2c0c0a-kube-api-access-lf8gj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:50:59.590047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:59.590020 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" Nov 03 16:50:59.590614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:59.590051 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg" event={"ID":"129681f8-da05-4f1f-978b-7998ab2c0c0a","Type":"ContainerDied","Data":"38a762b3fb2dcc9fbe1cbdf15bdc1fe9eb2b95b4994edc72b55d2b68644c8647"} Nov 03 16:50:59.590614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:59.590122 4539 scope.go:122] "RemoveContainer" containerID="c058f0dee5a1fd81c80154cb3eec67f9748a00b6a7fd6ee2bec29983dc7f5e5d" Nov 03 16:50:59.649461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:59.649388 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg"] Nov 03 16:50:59.659106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:50:59.659078 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vklbg"] Nov 03 16:51:00.469881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:00.469842 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" path="/var/lib/kubelet/pods/129681f8-da05-4f1f-978b-7998ab2c0c0a/volumes" Nov 03 16:51:00.543138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:00.543106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:00.692469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:00.692416 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:01.196753 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:01.196709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:02.623201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:02.623155 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:03.606134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:03.606097 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t"] Nov 03 16:51:03.606417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:03.606368 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" containerID="cri-o://5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3" gracePeriod=30 Nov 03 16:51:04.011606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:04.011520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:04.777817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:04.776063 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:05.572189 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:05.572155 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6"] Nov 03 16:51:05.573269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:05.573237 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:05.725586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:05.725552 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:06.136092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:06.136032 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:06.291366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:06.291336 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:06.497965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:06.497880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:08.051049 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.051002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:08.883021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.882945 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:51:08.929409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.929372 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4"] Nov 03 16:51:08.976612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976578 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.976826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976711 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.976826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976756 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.976826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976795 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.977004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976840 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.977004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976891 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:08.977004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976924 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.977004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.976982 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.977205 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977075 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.977205 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977112 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:08.977305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977212 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy\") pod \"10caf43a-bae3-4650-93a2-e406713a73ca\" (UID: \"10caf43a-bae3-4650-93a2-e406713a73ca\") " Nov 03 16:51:08.977360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977297 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:08.977360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977305 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:08.977710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977689 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:08.977788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977717 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:08.977788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977730 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:08.977788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.977741 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:08.978552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.978522 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:08.979538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.979505 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:08.979740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.979717 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:08.979965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.979948 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:08.980147 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:08.980129 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd" pod "10caf43a-bae3-4650-93a2-e406713a73ca" (UID: "10caf43a-bae3-4650-93a2-e406713a73ca"). InnerVolumeSpecName "kube-api-access-2sqdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:09.034705 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.034661 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:09.078652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.078618 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10caf43a-bae3-4650-93a2-e406713a73ca-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.078652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.078647 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10caf43a-bae3-4650-93a2-e406713a73ca-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.078652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.078656 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10caf43a-bae3-4650-93a2-e406713a73ca-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.079293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.078666 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.079293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.078680 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqdd\" (UniqueName: \"kubernetes.io/projected/10caf43a-bae3-4650-93a2-e406713a73ca-kube-api-access-2sqdd\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.665834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.665800 4539 generic.go:356] "Generic (PLEG): container finished" podID="10caf43a-bae3-4650-93a2-e406713a73ca" containerID="5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3" exitCode=0 Nov 03 16:51:09.666032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.665856 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" Nov 03 16:51:09.666032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.665887 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" event={"ID":"10caf43a-bae3-4650-93a2-e406713a73ca","Type":"ContainerDied","Data":"5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3"} Nov 03 16:51:09.666032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.665944 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t" event={"ID":"10caf43a-bae3-4650-93a2-e406713a73ca","Type":"ContainerDied","Data":"7299e31f58cbe6fecc0802a9270edad63a02df5bc5364f1835a692a4b97a9b16"} Nov 03 16:51:09.666032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.665979 4539 scope.go:122] "RemoveContainer" containerID="5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3" Nov 03 16:51:09.678292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.678270 4539 scope.go:122] "RemoveContainer" containerID="5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3" Nov 03 16:51:09.678691 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:09.678667 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3\": container with ID starting with 5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3 not found: ID does not exist" containerID="5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3" Nov 03 16:51:09.678803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.678701 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3"} err="failed to get container status \"5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3\": rpc error: code = NotFound desc = could not find container \"5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3\": container with ID starting with 5c6f7e9233e86704843ba1fab7021a58932cd0adaa5767f530fe49ab135609a3 not found: ID does not exist" Nov 03 16:51:09.724755 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.724724 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t"] Nov 03 16:51:09.736739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.736703 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-6wf7t"] Nov 03 16:51:09.776645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:09.776609 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:10.469918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:10.469886 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" path="/var/lib/kubelet/pods/10caf43a-bae3-4650-93a2-e406713a73ca/volumes" Nov 03 16:51:11.419586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:11.419544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:11.748944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:11.748823 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:13.050177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:13.049886 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:14.298721 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.298688 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg"] Nov 03 16:51:14.299272 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.299257 4539 cpu_manager.go:414] "containerMap: removing container" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" Nov 03 16:51:14.299316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.299273 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" Nov 03 16:51:14.299316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.299311 4539 cpu_manager.go:414] "containerMap: removing container" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" Nov 03 16:51:14.299379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.299316 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" Nov 03 16:51:14.299414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.299401 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="10caf43a-bae3-4650-93a2-e406713a73ca" containerName="istio-proxy" Nov 03 16:51:14.299414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.299413 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="129681f8-da05-4f1f-978b-7998ab2c0c0a" containerName="istio-proxy" Nov 03 16:51:14.303241 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.303217 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.316676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.316653 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b9e362-openshift-default-dockercfg-mrk2t\"" Nov 03 16:51:14.332506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.332475 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg"] Nov 03 16:51:14.450889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.450852 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.450923 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451039 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451101 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdm7w\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451123 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451148 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451347 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451450 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.451508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.451497 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.552932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.552842 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.552932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.552920 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553005 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553077 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553176 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553169 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553197 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553237 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdm7w\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553283 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553312 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553502 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553653 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553826 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.553980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.553961 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.554167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.554149 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.556231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.556209 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.556359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.556339 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.592981 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.592941 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.594152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.594115 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdm7w\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.617929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.617883 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:14.942025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.941996 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg"] Nov 03 16:51:14.950971 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.950921 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:51:14.951214 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.951185 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:51:14.951335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:14.951244 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:51:15.052468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:15.049763 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:15.740912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:15.740871 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" event={"ID":"72285235-afd8-4fec-87ea-88b01e1d3161","Type":"ContainerStarted","Data":"44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639"} Nov 03 16:51:15.741413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:15.740934 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" event={"ID":"72285235-afd8-4fec-87ea-88b01e1d3161","Type":"ContainerStarted","Data":"4601216dfdc775f97758980ab2f2ed4fab651f200e7c9b26f7a5a9021b730e8f"} Nov 03 16:51:15.835450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:15.833909 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podStartSLOduration=1.8338874299999999 podStartE2EDuration="1.83388743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:15.796916014 +0000 UTC m=+2219.490187037" watchObservedRunningTime="2026-11-03 16:51:15.83388743 +0000 UTC m=+2219.527158453" Nov 03 16:51:16.441941 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.441903 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d"] Nov 03 16:51:16.447494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.447468 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.475850 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.475813 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d"] Nov 03 16:51:16.476817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.476795 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f2923b-serving-cert\"" Nov 03 16:51:16.557652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.557620 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:16.588931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.588896 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.589150 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.589123 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67dlq\" (UniqueName: \"kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.589394 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.589360 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.589810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.589607 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.641448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.640779 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:16.653261 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.652859 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 16:51:16.653261 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.652917 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 16:51:16.691318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.690305 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.691318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.690390 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.691318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.690439 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.691318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.690514 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67dlq\" (UniqueName: \"kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.692177 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:16.691874 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-serving-cert: secret "maas-api-e2e-praxis-b-f2923b-serving-cert" not found Nov 03 16:51:16.692177 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:16.691960 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls podName:a18d1788-acbf-4a39-8b9d-f59325436d16 nodeName:}" failed. No retries permitted until 2026-11-03 16:51:17.191936676 +0000 UTC m=+2220.885207692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls") pod "maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" (UID: "a18d1788-acbf-4a39-8b9d-f59325436d16") : secret "maas-api-e2e-praxis-b-f2923b-serving-cert" not found Nov 03 16:51:16.692788 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.692645 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.697960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.697930 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.715325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.715170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:16.725082 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.725013 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67dlq\" (UniqueName: \"kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:16.855842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.855814 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:16.960206 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.960105 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d"] Nov 03 16:51:16.960392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:16.960357 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" containerID="cri-o://b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264" gracePeriod=30 Nov 03 16:51:17.197107 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.197078 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:17.199753 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.199724 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:17.363186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.363150 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:17.697740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.679556 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 16:51:17.697740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.679615 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 16:51:17.780704 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.780668 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d"] Nov 03 16:51:17.783251 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:51:17.783220 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18d1788_acbf_4a39_8b9d_f59325436d16.slice/crio-f006bb738352bf51caf5fa1a37bb318dc3c72cd5e1c0cad3b7412f18f154d826 WatchSource:0}: Error finding container f006bb738352bf51caf5fa1a37bb318dc3c72cd5e1c0cad3b7412f18f154d826: Status 404 returned error can't find the container with id f006bb738352bf51caf5fa1a37bb318dc3c72cd5e1c0cad3b7412f18f154d826 Nov 03 16:51:17.785517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:17.785495 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:51:18.322401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:18.322309 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:18.621014 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:18.620586 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 16:51:18.621014 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:18.620642 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 16:51:18.781359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:18.781326 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" event={"ID":"a18d1788-acbf-4a39-8b9d-f59325436d16","Type":"ContainerStarted","Data":"744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e"} Nov 03 16:51:18.781616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:18.781383 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" event={"ID":"a18d1788-acbf-4a39-8b9d-f59325436d16","Type":"ContainerStarted","Data":"f006bb738352bf51caf5fa1a37bb318dc3c72cd5e1c0cad3b7412f18f154d826"} Nov 03 16:51:18.781616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:18.781483 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:19.619449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:19.618840 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 16:51:19.619449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:19.618906 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 16:51:20.047450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.034722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:20.596166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.596128 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" podUID="ba167e1b-d5ea-4874-a11c-6a09b710bd8b" containerName="main" containerID="cri-o://9ab6d890d5f879682535c8121257cd4de5bcab5edb302330c7a928ceb1e56b85" gracePeriod=45 Nov 03 16:51:20.622449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.618829 4539 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 16:51:20.622449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.618890 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 16:51:20.797066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.797033 4539 generic.go:356] "Generic (PLEG): container finished" podID="ba167e1b-d5ea-4874-a11c-6a09b710bd8b" containerID="9ab6d890d5f879682535c8121257cd4de5bcab5edb302330c7a928ceb1e56b85" exitCode=0 Nov 03 16:51:20.797247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.797197 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" event={"ID":"ba167e1b-d5ea-4874-a11c-6a09b710bd8b","Type":"ContainerDied","Data":"9ab6d890d5f879682535c8121257cd4de5bcab5edb302330c7a928ceb1e56b85"} Nov 03 16:51:20.975207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:20.975181 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:51:21.018489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.017947 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" podStartSLOduration=4.572408293 podStartE2EDuration="5.017925425s" totalImagesPullingTime="445.517132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:18.837342062 +0000 UTC m=+2222.530613085" watchObservedRunningTime="2026-11-03 16:51:21.017925425 +0000 UTC m=+2224.711196448" Nov 03 16:51:21.081505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081469 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir\") pod \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " Nov 03 16:51:21.081505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081522 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs\") pod \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " Nov 03 16:51:21.081736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081588 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache\") pod \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " Nov 03 16:51:21.081736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081631 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home\") pod \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " Nov 03 16:51:21.081736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081651 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv\" (UniqueName: \"kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv\") pod \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " Nov 03 16:51:21.081736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081684 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm\") pod \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\" (UID: \"ba167e1b-d5ea-4874-a11c-6a09b710bd8b\") " Nov 03 16:51:21.081958 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.081800 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir" pod "ba167e1b-d5ea-4874-a11c-6a09b710bd8b" (UID: "ba167e1b-d5ea-4874-a11c-6a09b710bd8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:21.082035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.082006 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache" pod "ba167e1b-d5ea-4874-a11c-6a09b710bd8b" (UID: "ba167e1b-d5ea-4874-a11c-6a09b710bd8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:21.082270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.082222 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:21.082270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.082247 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:21.082846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.082797 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home" pod "ba167e1b-d5ea-4874-a11c-6a09b710bd8b" (UID: "ba167e1b-d5ea-4874-a11c-6a09b710bd8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:21.084307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.084278 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm" pod "ba167e1b-d5ea-4874-a11c-6a09b710bd8b" (UID: "ba167e1b-d5ea-4874-a11c-6a09b710bd8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:21.084620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.084600 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs" pod "ba167e1b-d5ea-4874-a11c-6a09b710bd8b" (UID: "ba167e1b-d5ea-4874-a11c-6a09b710bd8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:21.084694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.084610 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv" pod "ba167e1b-d5ea-4874-a11c-6a09b710bd8b" (UID: "ba167e1b-d5ea-4874-a11c-6a09b710bd8b"). InnerVolumeSpecName "kube-api-access-tlmrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:21.183333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.183296 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:21.183333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.183326 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:21.183333 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.183336 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmrv\" (UniqueName: \"kubernetes.io/projected/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-kube-api-access-tlmrv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:21.183603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.183344 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba167e1b-d5ea-4874-a11c-6a09b710bd8b-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:21.435411 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.435345 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:21.668915 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.668879 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:21.673080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.670417 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:21.673080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.672848 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:51:21.880455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.869730 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" Nov 03 16:51:21.880455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.871563 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6" event={"ID":"ba167e1b-d5ea-4874-a11c-6a09b710bd8b","Type":"ContainerDied","Data":"9d8d2f15438a98ac83a5475380abc1c8c4ee84dfba700ea39bc79c6e75e67392"} Nov 03 16:51:21.880455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.871618 4539 scope.go:122] "RemoveContainer" containerID="9ab6d890d5f879682535c8121257cd4de5bcab5edb302330c7a928ceb1e56b85" Nov 03 16:51:21.952406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.952361 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6"] Nov 03 16:51:21.973860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:21.970034 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-b9e362/praxis-route-b9e362-kserve-5dfc6c886c-t4km6"] Nov 03 16:51:22.474242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.474204 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba167e1b-d5ea-4874-a11c-6a09b710bd8b" path="/var/lib/kubelet/pods/ba167e1b-d5ea-4874-a11c-6a09b710bd8b/volumes" Nov 03 16:51:22.545824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.545791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:22.688963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.688931 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:51:22.834998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.834966 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835027 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835092 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835139 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835263 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835295 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835335 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.835395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835351 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835400 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835455 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo\") pod \"5586617a-e287-4fdd-9d22-ad436f7f385c\" (UID: \"5586617a-e287-4fdd-9d22-ad436f7f385c\") " Nov 03 16:51:22.835620 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835493 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.835769 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835703 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.835827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.835775 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.836331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.836172 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.836600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.836576 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.836730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.836604 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.836730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.836619 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.837064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.837040 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:22.838055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.838018 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:22.838055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.838039 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.838223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.838192 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "kube-api-access-rsr6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:22.838621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.838584 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token" pod "5586617a-e287-4fdd-9d22-ad436f7f385c" (UID: "5586617a-e287-4fdd-9d22-ad436f7f385c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:22.886075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.886037 4539 generic.go:356] "Generic (PLEG): container finished" podID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerID="b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264" exitCode=0 Nov 03 16:51:22.886235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.886106 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" Nov 03 16:51:22.886235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.886130 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" event={"ID":"5586617a-e287-4fdd-9d22-ad436f7f385c","Type":"ContainerDied","Data":"b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264"} Nov 03 16:51:22.886235 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.886221 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d" event={"ID":"5586617a-e287-4fdd-9d22-ad436f7f385c","Type":"ContainerDied","Data":"0fa0182f68ded1a02e9ba1a7731c3bb45d89b06b683a88793d60011ecab87521"} Nov 03 16:51:22.886377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.886250 4539 scope.go:122] "RemoveContainer" containerID="b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264" Nov 03 16:51:22.897710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.897686 4539 scope.go:122] "RemoveContainer" containerID="b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264" Nov 03 16:51:22.898048 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:22.898028 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264\": container with ID starting with b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264 not found: ID does not exist" containerID="b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264" Nov 03 16:51:22.898094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.898059 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264"} err="failed to get container status \"b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264\": rpc error: code = NotFound desc = could not find container \"b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264\": container with ID starting with b3bab8e4496aef41044333bdd995386c257e2d22144764a2167faf49ca8df264 not found: ID does not exist" Nov 03 16:51:22.936461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.936416 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d"] Nov 03 16:51:22.937801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.937771 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5586617a-e287-4fdd-9d22-ad436f7f385c-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.937801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.937800 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.938016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.937809 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.938016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.937819 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5586617a-e287-4fdd-9d22-ad436f7f385c-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.938016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.937829 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsr6k\" (UniqueName: \"kubernetes.io/projected/5586617a-e287-4fdd-9d22-ad436f7f385c-kube-api-access-rsr6k\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.942388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:22.942359 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fjn4d"] Nov 03 16:51:23.954376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:23.954338 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" podUID="1a3e0dcd-871f-4076-bd65-4c6f99d95201" containerName="main" containerID="cri-o://8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696" gracePeriod=45 Nov 03 16:51:24.206552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.206415 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:24.262129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.262091 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:51:24.357352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357320 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache\") pod \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " Nov 03 16:51:24.357556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357382 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm\") pod \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " Nov 03 16:51:24.357556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357550 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home\") pod \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " Nov 03 16:51:24.357698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357614 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf\" (UniqueName: \"kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf\") pod \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " Nov 03 16:51:24.357698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357667 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache" pod "1a3e0dcd-871f-4076-bd65-4c6f99d95201" (UID: "1a3e0dcd-871f-4076-bd65-4c6f99d95201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.357823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357693 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir\") pod \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " Nov 03 16:51:24.357823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357738 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs\") pod \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\" (UID: \"1a3e0dcd-871f-4076-bd65-4c6f99d95201\") " Nov 03 16:51:24.357921 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.357900 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home" pod "1a3e0dcd-871f-4076-bd65-4c6f99d95201" (UID: "1a3e0dcd-871f-4076-bd65-4c6f99d95201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.358367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.358343 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.358367 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.358368 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.358944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.358909 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir" pod "1a3e0dcd-871f-4076-bd65-4c6f99d95201" (UID: "1a3e0dcd-871f-4076-bd65-4c6f99d95201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.360650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.360619 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs" pod "1a3e0dcd-871f-4076-bd65-4c6f99d95201" (UID: "1a3e0dcd-871f-4076-bd65-4c6f99d95201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:24.360840 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.360815 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm" pod "1a3e0dcd-871f-4076-bd65-4c6f99d95201" (UID: "1a3e0dcd-871f-4076-bd65-4c6f99d95201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.361339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.361311 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf" pod "1a3e0dcd-871f-4076-bd65-4c6f99d95201" (UID: "1a3e0dcd-871f-4076-bd65-4c6f99d95201"). InnerVolumeSpecName "kube-api-access-796lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:24.459935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.459837 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.459935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.459873 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e0dcd-871f-4076-bd65-4c6f99d95201-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.459935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.459885 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e0dcd-871f-4076-bd65-4c6f99d95201-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.459935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.459896 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796lf\" (UniqueName: \"kubernetes.io/projected/1a3e0dcd-871f-4076-bd65-4c6f99d95201-kube-api-access-796lf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.469758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.469727 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" path="/var/lib/kubelet/pods/5586617a-e287-4fdd-9d22-ad436f7f385c/volumes" Nov 03 16:51:24.794315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.794290 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:51:24.902743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.902709 4539 generic.go:356] "Generic (PLEG): container finished" podID="1a3e0dcd-871f-4076-bd65-4c6f99d95201" containerID="8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696" exitCode=0 Nov 03 16:51:24.902946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.902758 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" Nov 03 16:51:24.902946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.902793 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" event={"ID":"1a3e0dcd-871f-4076-bd65-4c6f99d95201","Type":"ContainerDied","Data":"8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696"} Nov 03 16:51:24.902946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.902848 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4" event={"ID":"1a3e0dcd-871f-4076-bd65-4c6f99d95201","Type":"ContainerDied","Data":"d0342df1cf8c6c6790695359a76e5b20288a38baa57e88ff07cd05e4843e53a5"} Nov 03 16:51:24.902946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.902866 4539 scope.go:122] "RemoveContainer" containerID="8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696" Nov 03 16:51:24.914853 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.914831 4539 scope.go:122] "RemoveContainer" containerID="8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696" Nov 03 16:51:24.915204 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:24.915182 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696\": container with ID starting with 8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696 not found: ID does not exist" containerID="8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696" Nov 03 16:51:24.915300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.915218 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696"} err="failed to get container status \"8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696\": rpc error: code = NotFound desc = could not find container \"8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696\": container with ID starting with 8a45aeb3d6673e90888a7c9482d7e52b5dec25436ca83e68fb1d9804fe222696 not found: ID does not exist" Nov 03 16:51:24.955418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.955386 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4"] Nov 03 16:51:24.958694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:24.958671 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-9947ab/ipp-route-9947ab-kserve-6bf9ffdfc-5n6k4"] Nov 03 16:51:25.475649 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:25.475613 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:25.639376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:25.639341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:25.920312 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:25.920281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:26.156909 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:26.156874 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:51:26.311530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:26.311498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:26.469703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:26.469668 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3e0dcd-871f-4076-bd65-4c6f99d95201" path="/var/lib/kubelet/pods/1a3e0dcd-871f-4076-bd65-4c6f99d95201/volumes" Nov 03 16:51:26.645926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:26.645830 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6"] Nov 03 16:51:26.646112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:26.646072 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" containerID="cri-o://2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a" gracePeriod=30 Nov 03 16:51:26.726046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:26.726010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:27.529867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:27.529820 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:29.674598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:29.674563 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:30.703973 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:30.703936 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:31.932925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.932895 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:51:31.962362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.962334 4539 generic.go:356] "Generic (PLEG): container finished" podID="f6167316-2853-40c1-9357-1b10c4b471a5" containerID="2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a" exitCode=0 Nov 03 16:51:31.962555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.962386 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" Nov 03 16:51:31.962555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.962433 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" event={"ID":"f6167316-2853-40c1-9357-1b10c4b471a5","Type":"ContainerDied","Data":"2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a"} Nov 03 16:51:31.962555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.962486 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6" event={"ID":"f6167316-2853-40c1-9357-1b10c4b471a5","Type":"ContainerDied","Data":"42ba454dbcc072df5e79bd36f37826f5fe62ea4482a62ae6c55c3ab42513b946"} Nov 03 16:51:31.962555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.962513 4539 scope.go:122] "RemoveContainer" containerID="2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a" Nov 03 16:51:31.972702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.972684 4539 scope.go:122] "RemoveContainer" containerID="b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370" Nov 03 16:51:31.981725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.981709 4539 scope.go:122] "RemoveContainer" containerID="2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a" Nov 03 16:51:31.981954 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:31.981928 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a\": container with ID starting with 2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a not found: ID does not exist" containerID="2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a" Nov 03 16:51:31.982028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.981957 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a"} err="failed to get container status \"2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a\": rpc error: code = NotFound desc = could not find container \"2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a\": container with ID starting with 2e9810cbb70c044580030e38b9bd4e2615845150cc1df0d24c64297cc2aec32a not found: ID does not exist" Nov 03 16:51:31.982028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.981975 4539 scope.go:122] "RemoveContainer" containerID="b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370" Nov 03 16:51:31.982268 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:31.982250 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370\": container with ID starting with b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370 not found: ID does not exist" containerID="b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370" Nov 03 16:51:31.982318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:31.982277 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370"} err="failed to get container status \"b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370\": rpc error: code = NotFound desc = could not find container \"b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370\": container with ID starting with b2a50089a4c153050a87e78eb4a29fbd3fa448340c1426200d0a62f045ff7370 not found: ID does not exist" Nov 03 16:51:32.065237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065203 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065268 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065309 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065335 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065471 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065516 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065582 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065643 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065684 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket\") pod \"f6167316-2853-40c1-9357-1b10c4b471a5\" (UID: \"f6167316-2853-40c1-9357-1b10c4b471a5\") " Nov 03 16:51:32.065879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065805 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:32.065986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065874 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:32.065986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.065894 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:32.066095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066034 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:32.066095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066064 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:32.066369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066272 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.066369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066284 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6167316-2853-40c1-9357-1b10c4b471a5-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.066369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066293 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.066369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066300 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.066369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.066310 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.067862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.067841 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:32.067966 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.067845 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "kube-api-access-xvzkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:32.068219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.068198 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:32.068269 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.068198 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo" pod "f6167316-2853-40c1-9357-1b10c4b471a5" (UID: "f6167316-2853-40c1-9357-1b10c4b471a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:32.167314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.167242 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvzkk\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-kube-api-access-xvzkk\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.167314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.167266 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6167316-2853-40c1-9357-1b10c4b471a5-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.167314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.167275 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6167316-2853-40c1-9357-1b10c4b471a5-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.167314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.167283 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6167316-2853-40c1-9357-1b10c4b471a5-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:32.306046 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.306007 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6"] Nov 03 16:51:32.316495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.316468 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6bb42-openshift-default-6f9c779dc7-bm8s6"] Nov 03 16:51:32.468704 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.468630 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" path="/var/lib/kubelet/pods/f6167316-2853-40c1-9357-1b10c4b471a5/volumes" Nov 03 16:51:32.743841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:32.743737 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:33.712990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:33.712955 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:34.045619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:34.045588 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:34.190379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:34.190339 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:34.458839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:34.458753 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:35.742774 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:35.742732 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr"] Nov 03 16:51:35.743289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:35.743010 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" podUID="7653bca9-7e28-4343-beff-c04c7e9b8be3" containerName="maas-api" containerID="cri-o://589c9fd3ad5daa722bf4969052d8fdd2443295fd02ccc11af260772be8ce7f8f" gracePeriod=30 Nov 03 16:51:35.835541 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:35.835512 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:36.070449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.070361 4539 generic.go:356] "Generic (PLEG): container finished" podID="7653bca9-7e28-4343-beff-c04c7e9b8be3" containerID="589c9fd3ad5daa722bf4969052d8fdd2443295fd02ccc11af260772be8ce7f8f" exitCode=0 Nov 03 16:51:36.071783 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.071717 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" event={"ID":"7653bca9-7e28-4343-beff-c04c7e9b8be3","Type":"ContainerDied","Data":"589c9fd3ad5daa722bf4969052d8fdd2443295fd02ccc11af260772be8ce7f8f"} Nov 03 16:51:36.307873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.307848 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:51:36.351817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.351702 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\") pod \"7653bca9-7e28-4343-beff-c04c7e9b8be3\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " Nov 03 16:51:36.351817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.351765 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm\" (UniqueName: \"kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm\") pod \"7653bca9-7e28-4343-beff-c04c7e9b8be3\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " Nov 03 16:51:36.352090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.351824 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca\") pod \"7653bca9-7e28-4343-beff-c04c7e9b8be3\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " Nov 03 16:51:36.352090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.351949 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs\") pod \"7653bca9-7e28-4343-beff-c04c7e9b8be3\" (UID: \"7653bca9-7e28-4343-beff-c04c7e9b8be3\") " Nov 03 16:51:36.354998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.354963 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm" pod "7653bca9-7e28-4343-beff-c04c7e9b8be3" (UID: "7653bca9-7e28-4343-beff-c04c7e9b8be3"). InnerVolumeSpecName "kube-api-access-rzdhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:36.355690 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.355658 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca" pod "7653bca9-7e28-4343-beff-c04c7e9b8be3" (UID: "7653bca9-7e28-4343-beff-c04c7e9b8be3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:36.358006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.357959 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls" pod "7653bca9-7e28-4343-beff-c04c7e9b8be3" (UID: "7653bca9-7e28-4343-beff-c04c7e9b8be3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:36.359832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.359808 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs" pod "7653bca9-7e28-4343-beff-c04c7e9b8be3" (UID: "7653bca9-7e28-4343-beff-c04c7e9b8be3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:36.453404 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.453363 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:36.453404 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.453393 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdhm\" (UniqueName: \"kubernetes.io/projected/7653bca9-7e28-4343-beff-c04c7e9b8be3-kube-api-access-rzdhm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:36.453404 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.453404 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7653bca9-7e28-4343-beff-c04c7e9b8be3-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:36.453404 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.453412 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7653bca9-7e28-4343-beff-c04c7e9b8be3-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:51:36.589695 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:36.589657 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:37.079629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:37.079596 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" Nov 03 16:51:37.080080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:37.079641 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr" event={"ID":"7653bca9-7e28-4343-beff-c04c7e9b8be3","Type":"ContainerDied","Data":"f3573ffa5beaee0630339c229acd6d84efe74067ba5fd8060906ba00d2365a17"} Nov 03 16:51:37.080080 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:37.079700 4539 scope.go:122] "RemoveContainer" containerID="589c9fd3ad5daa722bf4969052d8fdd2443295fd02ccc11af260772be8ce7f8f" Nov 03 16:51:37.129329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:37.129296 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr"] Nov 03 16:51:37.139382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:37.139355 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-31de0f-c48cd9bbb-rqrjr"] Nov 03 16:51:38.180794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:38.180756 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:38.469185 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:38.469109 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7653bca9-7e28-4343-beff-c04c7e9b8be3" path="/var/lib/kubelet/pods/7653bca9-7e28-4343-beff-c04c7e9b8be3/volumes" Nov 03 16:51:40.060088 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:40.060050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:40.485385 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:40.485297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:40.627772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:40.627739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:41.213871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:41.213800 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:41.578569 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:41.578536 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:42.553259 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:42.553224 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:42.701007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:42.700957 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:43.160016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:43.159982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:43.772669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:43.772639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:44.459036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:44.459000 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:45.314315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:45.314284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:46.194266 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:46.194225 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:47.493138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:47.493098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:48.074931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:48.074894 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:48.478930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:48.478847 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:49.693213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:49.693102 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:49.973381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:49.973299 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:51.410962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:51.410908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:52.594602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:52.594566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:53.416909 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:53.416867 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:53.615285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:53.615248 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:54.767826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:54.767792 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:55.832676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:55.832645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:56.052083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:56.052048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:56.158166 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:51:56.158074 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:51:57.402722 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:57.402617 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:57.942532 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:57.942494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:58.104879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:58.104844 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:58.949213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:58.949173 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:59.290796 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:59.290762 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:51:59.532725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:51:59.532680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:01.099294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:01.099254 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:02.724363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:02.724318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:04.787747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:04.787652 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:05.198209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:05.198112 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:05.831969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:05.831934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:06.674777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:06.674746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:08.356968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:08.356931 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:09.991996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:09.991948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:11.215089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:11.215048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:11.648834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:11.648081 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:11.910766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:11.910626 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:13.090677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.090638 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l"] Nov 03 16:52:13.091192 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.090939 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" podUID="8d4092e9-0202-4395-809c-fa9bad375cf3" containerName="maas-api" containerID="cri-o://aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77" gracePeriod=30 Nov 03 16:52:13.193942 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.193907 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:13.422195 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.422157 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:13.442154 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.442112 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:52:13.445027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.444994 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca\") pod \"8d4092e9-0202-4395-809c-fa9bad375cf3\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " Nov 03 16:52:13.445223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.445074 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\") pod \"8d4092e9-0202-4395-809c-fa9bad375cf3\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " Nov 03 16:52:13.445223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.445128 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df\" (UniqueName: \"kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df\") pod \"8d4092e9-0202-4395-809c-fa9bad375cf3\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " Nov 03 16:52:13.445223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.445209 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs\") pod \"8d4092e9-0202-4395-809c-fa9bad375cf3\" (UID: \"8d4092e9-0202-4395-809c-fa9bad375cf3\") " Nov 03 16:52:13.445525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.445502 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca" pod "8d4092e9-0202-4395-809c-fa9bad375cf3" (UID: "8d4092e9-0202-4395-809c-fa9bad375cf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:13.445768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.445747 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d4092e9-0202-4395-809c-fa9bad375cf3-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:13.446895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.446861 4539 generic.go:356] "Generic (PLEG): container finished" podID="8d4092e9-0202-4395-809c-fa9bad375cf3" containerID="aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77" exitCode=0 Nov 03 16:52:13.447178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.447149 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" event={"ID":"8d4092e9-0202-4395-809c-fa9bad375cf3","Type":"ContainerDied","Data":"aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77"} Nov 03 16:52:13.447278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.447218 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" event={"ID":"8d4092e9-0202-4395-809c-fa9bad375cf3","Type":"ContainerDied","Data":"86198c57319b38cc992b8fdaef0d1aba5b18fbddc69d2283b763fafda70494d3"} Nov 03 16:52:13.447278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.447250 4539 scope.go:122] "RemoveContainer" containerID="aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77" Nov 03 16:52:13.451801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.451768 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs" pod "8d4092e9-0202-4395-809c-fa9bad375cf3" (UID: "8d4092e9-0202-4395-809c-fa9bad375cf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:13.453929 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.453901 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls" pod "8d4092e9-0202-4395-809c-fa9bad375cf3" (UID: "8d4092e9-0202-4395-809c-fa9bad375cf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:13.458325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.458297 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df" pod "8d4092e9-0202-4395-809c-fa9bad375cf3" (UID: "8d4092e9-0202-4395-809c-fa9bad375cf3"). InnerVolumeSpecName "kube-api-access-4d7df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:13.489086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.489059 4539 scope.go:122] "RemoveContainer" containerID="aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77" Nov 03 16:52:13.489512 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:13.489478 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77\": container with ID starting with aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77 not found: ID does not exist" containerID="aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77" Nov 03 16:52:13.489658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.489525 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77"} err="failed to get container status \"aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77\": rpc error: code = NotFound desc = could not find container \"aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77\": container with ID starting with aa1df6eaed312301690204bdf3ebdbc1dccd5074103595440913cb7c33af1d77 not found: ID does not exist" Nov 03 16:52:13.546571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.546528 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:13.547594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.546798 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d7df\" (UniqueName: \"kubernetes.io/projected/8d4092e9-0202-4395-809c-fa9bad375cf3-kube-api-access-4d7df\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:13.547594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.546824 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4092e9-0202-4395-809c-fa9bad375cf3-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:13.647152 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:13.646941 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:14.099245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:14.099208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:14.455627 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:14.455546 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l" Nov 03 16:52:14.508772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:14.508730 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l"] Nov 03 16:52:14.523350 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:14.523323 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9947ab-6b74d789c4-qdg5l"] Nov 03 16:52:15.928491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:15.928456 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq"] Nov 03 16:52:15.929028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:15.928722 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" podUID="0641c0c8-4413-4dbc-a46e-8612fda4f05b" containerName="maas-api" containerID="cri-o://d63be1b7ebe6c4a7662b6723ad42a55bfa01862405e924fc8f4713bc7fd41a44" gracePeriod=30 Nov 03 16:52:16.029279 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.029233 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:16.406714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.406678 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp"] Nov 03 16:52:16.470412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.450666 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:16.470412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.468881 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" podUID="eea3c944-079d-4f52-955b-bbd2f4d9e311" containerName="maas-api" containerID="cri-o://0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098" gracePeriod=30 Nov 03 16:52:16.489943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.489868 4539 generic.go:356] "Generic (PLEG): container finished" podID="0641c0c8-4413-4dbc-a46e-8612fda4f05b" containerID="d63be1b7ebe6c4a7662b6723ad42a55bfa01862405e924fc8f4713bc7fd41a44" exitCode=0 Nov 03 16:52:16.495140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.493826 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:52:16.495140 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.494092 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4092e9-0202-4395-809c-fa9bad375cf3" path="/var/lib/kubelet/pods/8d4092e9-0202-4395-809c-fa9bad375cf3/volumes" Nov 03 16:52:16.499539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.496389 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" event={"ID":"0641c0c8-4413-4dbc-a46e-8612fda4f05b","Type":"ContainerDied","Data":"d63be1b7ebe6c4a7662b6723ad42a55bfa01862405e924fc8f4713bc7fd41a44"} Nov 03 16:52:16.499539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.496485 4539 scope.go:122] "RemoveContainer" containerID="d63be1b7ebe6c4a7662b6723ad42a55bfa01862405e924fc8f4713bc7fd41a44" Nov 03 16:52:16.651208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.651177 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:16.653571 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.653545 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd\" (UniqueName: \"kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd\") pod \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " Nov 03 16:52:16.654232 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.654211 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs\") pod \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " Nov 03 16:52:16.655111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.655091 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls\") pod \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " Nov 03 16:52:16.656275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.656254 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca\") pod \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\" (UID: \"0641c0c8-4413-4dbc-a46e-8612fda4f05b\") " Nov 03 16:52:16.656901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.656790 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca" pod "0641c0c8-4413-4dbc-a46e-8612fda4f05b" (UID: "0641c0c8-4413-4dbc-a46e-8612fda4f05b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:16.659272 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.659243 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641c0c8-4413-4dbc-a46e-8612fda4f05b-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:16.663291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.663252 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs" pod "0641c0c8-4413-4dbc-a46e-8612fda4f05b" (UID: "0641c0c8-4413-4dbc-a46e-8612fda4f05b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:16.663712 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.663669 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd" pod "0641c0c8-4413-4dbc-a46e-8612fda4f05b" (UID: "0641c0c8-4413-4dbc-a46e-8612fda4f05b"). InnerVolumeSpecName "kube-api-access-jczkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:16.665994 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.665955 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls" pod "0641c0c8-4413-4dbc-a46e-8612fda4f05b" (UID: "0641c0c8-4413-4dbc-a46e-8612fda4f05b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:16.765247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.765196 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jczkd\" (UniqueName: \"kubernetes.io/projected/0641c0c8-4413-4dbc-a46e-8612fda4f05b-kube-api-access-jczkd\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:16.765247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.765250 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:16.765531 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.765266 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641c0c8-4413-4dbc-a46e-8612fda4f05b-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:16.810447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.807541 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:16.877873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.877844 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:52:16.968137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.968097 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca\") pod \"eea3c944-079d-4f52-955b-bbd2f4d9e311\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " Nov 03 16:52:16.969275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.969141 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca" pod "eea3c944-079d-4f52-955b-bbd2f4d9e311" (UID: "eea3c944-079d-4f52-955b-bbd2f4d9e311"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:16.969588 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.969566 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5\" (UniqueName: \"kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5\") pod \"eea3c944-079d-4f52-955b-bbd2f4d9e311\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " Nov 03 16:52:16.970553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.970535 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs\") pod \"eea3c944-079d-4f52-955b-bbd2f4d9e311\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " Nov 03 16:52:16.971144 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.971127 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\") pod \"eea3c944-079d-4f52-955b-bbd2f4d9e311\" (UID: \"eea3c944-079d-4f52-955b-bbd2f4d9e311\") " Nov 03 16:52:16.972583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.972387 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:16.972681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.972609 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5" pod "eea3c944-079d-4f52-955b-bbd2f4d9e311" (UID: "eea3c944-079d-4f52-955b-bbd2f4d9e311"). InnerVolumeSpecName "kube-api-access-c7pb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:16.972849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.972819 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pb5\" (UniqueName: \"kubernetes.io/projected/eea3c944-079d-4f52-955b-bbd2f4d9e311-kube-api-access-c7pb5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:16.972923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.972848 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea3c944-079d-4f52-955b-bbd2f4d9e311-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:16.974668 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.974629 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls" pod "eea3c944-079d-4f52-955b-bbd2f4d9e311" (UID: "eea3c944-079d-4f52-955b-bbd2f4d9e311"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:16.977522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:16.977482 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs" pod "eea3c944-079d-4f52-955b-bbd2f4d9e311" (UID: "eea3c944-079d-4f52-955b-bbd2f4d9e311"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:17.083455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.082732 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.083455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.082772 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea3c944-079d-4f52-955b-bbd2f4d9e311-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.499699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.499657 4539 generic.go:356] "Generic (PLEG): container finished" podID="eea3c944-079d-4f52-955b-bbd2f4d9e311" containerID="0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098" exitCode=0 Nov 03 16:52:17.499918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.499703 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" event={"ID":"eea3c944-079d-4f52-955b-bbd2f4d9e311","Type":"ContainerDied","Data":"0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098"} Nov 03 16:52:17.499918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.499758 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" event={"ID":"eea3c944-079d-4f52-955b-bbd2f4d9e311","Type":"ContainerDied","Data":"bafcf89fa658143657266509e3773ebd471b19a4dda84f37f5bbac659ce5770c"} Nov 03 16:52:17.499918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.499782 4539 scope.go:122] "RemoveContainer" containerID="0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098" Nov 03 16:52:17.499918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.499717 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp" Nov 03 16:52:17.499918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.499908 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" event={"ID":"0641c0c8-4413-4dbc-a46e-8612fda4f05b","Type":"ContainerDied","Data":"5a533598f3b50351e0e71a782c98a5f7d13dab1748f5fd370b14c31d2efdf14e"} Nov 03 16:52:17.500172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.500099 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq" Nov 03 16:52:17.513598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.513578 4539 scope.go:122] "RemoveContainer" containerID="0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098" Nov 03 16:52:17.513943 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:17.513918 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098\": container with ID starting with 0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098 not found: ID does not exist" containerID="0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098" Nov 03 16:52:17.514031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.513950 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098"} err="failed to get container status \"0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098\": rpc error: code = NotFound desc = could not find container \"0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098\": container with ID starting with 0ca0dd345b901697e7264f5c1d4e83e5780a285d694c033f7fd5fde4bfd7d098 not found: ID does not exist" Nov 03 16:52:17.588579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.588416 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq"] Nov 03 16:52:17.593496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.593467 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9e362-7cd6b8dc9c-pcvsq"] Nov 03 16:52:17.627999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.627940 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp"] Nov 03 16:52:17.634996 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.634962 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5a7d95-7bdb4458f7-l88zp"] Nov 03 16:52:17.715945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:17.715906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:18.470120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:18.470082 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0641c0c8-4413-4dbc-a46e-8612fda4f05b" path="/var/lib/kubelet/pods/0641c0c8-4413-4dbc-a46e-8612fda4f05b/volumes" Nov 03 16:52:18.470847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:18.470825 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea3c944-079d-4f52-955b-bbd2f4d9e311" path="/var/lib/kubelet/pods/eea3c944-079d-4f52-955b-bbd2f4d9e311/volumes" Nov 03 16:52:18.947694 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:18.947665 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:19.112054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:19.112020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:19.486097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:19.486061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:19.767830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:19.767734 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:20.261450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:20.258873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:21.036773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:21.036738 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:21.644963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:21.644921 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:22.672913 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:22.672880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:22.999360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:22.999272 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:23.556239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:23.556199 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:23.817594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:23.817503 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:23.905918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:23.905881 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg"] Nov 03 16:52:23.906527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:23.906497 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" containerID="cri-o://44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639" gracePeriod=30 Nov 03 16:52:24.029462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:24.028502 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:24.339153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:24.339119 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:24.764497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:24.764404 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:24.922141 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:24.922106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:25.513395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:25.513362 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:25.692612 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:25.692573 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:25.924480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:25.924418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:26.158172 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:26.158137 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:52:26.307028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:26.306988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:26.697308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:26.697219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:28.194009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:28.193948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:28.346121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:28.346083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:29.191767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.191699 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:52:29.273171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273133 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273199 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273282 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273333 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273396 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273460 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273497 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273597 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273729 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo\") pod \"72285235-afd8-4fec-87ea-88b01e1d3161\" (UID: \"72285235-afd8-4fec-87ea-88b01e1d3161\") " Nov 03 16:52:29.273998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.273867 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.274322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.274295 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.274967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.274388 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.275071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.274406 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.275071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.274646 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.275071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.274903 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:29.276193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.276164 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:29.276193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.276165 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.276316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.276204 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:29.276535 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.276517 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w" pod "72285235-afd8-4fec-87ea-88b01e1d3161" (UID: "72285235-afd8-4fec-87ea-88b01e1d3161"). InnerVolumeSpecName "kube-api-access-tdm7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:29.376455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376385 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72285235-afd8-4fec-87ea-88b01e1d3161-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376419 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdm7w\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-kube-api-access-tdm7w\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376463 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72285235-afd8-4fec-87ea-88b01e1d3161-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376471 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376479 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376487 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376495 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72285235-afd8-4fec-87ea-88b01e1d3161-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.376713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.376503 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72285235-afd8-4fec-87ea-88b01e1d3161-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.602253 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.602212 4539 generic.go:356] "Generic (PLEG): container finished" podID="72285235-afd8-4fec-87ea-88b01e1d3161" containerID="44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639" exitCode=0 Nov 03 16:52:29.602510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.602275 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" Nov 03 16:52:29.602510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.602300 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" event={"ID":"72285235-afd8-4fec-87ea-88b01e1d3161","Type":"ContainerDied","Data":"44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639"} Nov 03 16:52:29.602510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.602348 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg" event={"ID":"72285235-afd8-4fec-87ea-88b01e1d3161","Type":"ContainerDied","Data":"4601216dfdc775f97758980ab2f2ed4fab651f200e7c9b26f7a5a9021b730e8f"} Nov 03 16:52:29.602510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.602368 4539 scope.go:122] "RemoveContainer" containerID="44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639" Nov 03 16:52:29.615854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.615825 4539 scope.go:122] "RemoveContainer" containerID="44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639" Nov 03 16:52:29.616217 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:29.616186 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639\": container with ID starting with 44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639 not found: ID does not exist" containerID="44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639" Nov 03 16:52:29.616306 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.616226 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639"} err="failed to get container status \"44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639\": rpc error: code = NotFound desc = could not find container \"44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639\": container with ID starting with 44bc2b797d5e04de3d6d8336d98ef22821267e45bec917694dcec48e75488639 not found: ID does not exist" Nov 03 16:52:29.650860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.650826 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg"] Nov 03 16:52:29.661023 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.660989 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-gr7mg"] Nov 03 16:52:29.710517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.710479 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:29.860837 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:29.860747 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:30.468663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:30.468625 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" path="/var/lib/kubelet/pods/72285235-afd8-4fec-87ea-88b01e1d3161/volumes" Nov 03 16:52:30.568022 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:30.567986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:30.718566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:30.718530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:31.593212 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:31.593178 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:31.805011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:31.804965 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:31.992453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:31.992351 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:32.960998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:32.960967 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:33.546064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:33.546034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:33.890736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:33.890627 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:34.949568 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:34.949533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:35.138460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:35.138256 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:35.293287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:35.293257 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:35.881528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:35.881495 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:36.189572 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:36.189485 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:36.343952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:36.343911 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:37.536581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:37.536534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:37.974447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:37.974314 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:39.072167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:39.072135 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:39.671230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:39.669158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.027059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:41.027009 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.207846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:41.207529 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.353923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:41.353834 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:41.793207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:41.793175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:43.080286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.080251 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d"] Nov 03 16:52:43.080824 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.080556 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" podUID="a18d1788-acbf-4a39-8b9d-f59325436d16" containerName="maas-api" containerID="cri-o://744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e" gracePeriod=30 Nov 03 16:52:43.185293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.185253 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:43.528480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.523815 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:52:43.609045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.609013 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs\") pod \"a18d1788-acbf-4a39-8b9d-f59325436d16\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " Nov 03 16:52:43.609231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.609188 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq\" (UniqueName: \"kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq\") pod \"a18d1788-acbf-4a39-8b9d-f59325436d16\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " Nov 03 16:52:43.609297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.609251 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\") pod \"a18d1788-acbf-4a39-8b9d-f59325436d16\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " Nov 03 16:52:43.609357 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.609307 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca\") pod \"a18d1788-acbf-4a39-8b9d-f59325436d16\" (UID: \"a18d1788-acbf-4a39-8b9d-f59325436d16\") " Nov 03 16:52:43.610100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.609896 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca" pod "a18d1788-acbf-4a39-8b9d-f59325436d16" (UID: "a18d1788-acbf-4a39-8b9d-f59325436d16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:43.612120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.612034 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq" pod "a18d1788-acbf-4a39-8b9d-f59325436d16" (UID: "a18d1788-acbf-4a39-8b9d-f59325436d16"). InnerVolumeSpecName "kube-api-access-67dlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:43.612120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.612077 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs" pod "a18d1788-acbf-4a39-8b9d-f59325436d16" (UID: "a18d1788-acbf-4a39-8b9d-f59325436d16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:43.612256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.612178 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls" pod "a18d1788-acbf-4a39-8b9d-f59325436d16" (UID: "a18d1788-acbf-4a39-8b9d-f59325436d16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:43.710717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.710624 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:43.710717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.710654 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18d1788-acbf-4a39-8b9d-f59325436d16-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:43.710717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.710669 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18d1788-acbf-4a39-8b9d-f59325436d16-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:43.710717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.710680 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67dlq\" (UniqueName: \"kubernetes.io/projected/a18d1788-acbf-4a39-8b9d-f59325436d16-kube-api-access-67dlq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:43.711943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.711908 4539 generic.go:356] "Generic (PLEG): container finished" podID="a18d1788-acbf-4a39-8b9d-f59325436d16" containerID="744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e" exitCode=0 Nov 03 16:52:43.712084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.711953 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" Nov 03 16:52:43.712084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.711989 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" event={"ID":"a18d1788-acbf-4a39-8b9d-f59325436d16","Type":"ContainerDied","Data":"744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e"} Nov 03 16:52:43.712084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.712046 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d" event={"ID":"a18d1788-acbf-4a39-8b9d-f59325436d16","Type":"ContainerDied","Data":"f006bb738352bf51caf5fa1a37bb318dc3c72cd5e1c0cad3b7412f18f154d826"} Nov 03 16:52:43.712084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.712068 4539 scope.go:122] "RemoveContainer" containerID="744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e" Nov 03 16:52:43.723304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.723283 4539 scope.go:122] "RemoveContainer" containerID="744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e" Nov 03 16:52:43.723657 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:43.723638 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e\": container with ID starting with 744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e not found: ID does not exist" containerID="744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e" Nov 03 16:52:43.723727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.723669 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e"} err="failed to get container status \"744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e\": rpc error: code = NotFound desc = could not find container \"744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e\": container with ID starting with 744a7569fe7b7b45c781e21591253d87079b507392fa0914664dcd6ad924811e not found: ID does not exist" Nov 03 16:52:43.761940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.761900 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d"] Nov 03 16:52:43.766017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.765988 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2923b-5cfd5fdb49-zgl2d"] Nov 03 16:52:43.840795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.840755 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:43.979965 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:43.979875 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:44.469518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:44.469485 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18d1788-acbf-4a39-8b9d-f59325436d16" path="/var/lib/kubelet/pods/a18d1788-acbf-4a39-8b9d-f59325436d16/volumes" Nov 03 16:52:44.528328 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:44.528299 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:44.717509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:44.717475 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:45.288739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:45.288709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:45.710073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:45.709990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:45.884054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:45.884020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:46.040601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:46.040564 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:46.643376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:46.643341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:47.281474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.270803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:47.572802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.572710 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l"] Nov 03 16:52:47.573038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.573007 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" podUID="825917b2-9640-4c6a-bd63-d1adbe2e7631" containerName="maas-api" containerID="cri-o://966796f5259ca16206af42011785039f4f9d2a9baa1398f8a5a8784e3224766e" gracePeriod=30 Nov 03 16:52:47.681952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.681914 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:47.743515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.743479 4539 generic.go:356] "Generic (PLEG): container finished" podID="825917b2-9640-4c6a-bd63-d1adbe2e7631" containerID="966796f5259ca16206af42011785039f4f9d2a9baa1398f8a5a8784e3224766e" exitCode=0 Nov 03 16:52:47.743760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.743559 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" event={"ID":"825917b2-9640-4c6a-bd63-d1adbe2e7631","Type":"ContainerDied","Data":"966796f5259ca16206af42011785039f4f9d2a9baa1398f8a5a8784e3224766e"} Nov 03 16:52:47.931345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:47.931318 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:52:48.080819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.080779 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls\") pod \"825917b2-9640-4c6a-bd63-d1adbe2e7631\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " Nov 03 16:52:48.081017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.080832 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6\" (UniqueName: \"kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6\") pod \"825917b2-9640-4c6a-bd63-d1adbe2e7631\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " Nov 03 16:52:48.081017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.080979 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs\") pod \"825917b2-9640-4c6a-bd63-d1adbe2e7631\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " Nov 03 16:52:48.081114 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.081043 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca\") pod \"825917b2-9640-4c6a-bd63-d1adbe2e7631\" (UID: \"825917b2-9640-4c6a-bd63-d1adbe2e7631\") " Nov 03 16:52:48.081475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.081441 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca" pod "825917b2-9640-4c6a-bd63-d1adbe2e7631" (UID: "825917b2-9640-4c6a-bd63-d1adbe2e7631"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:48.081823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.081778 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/825917b2-9640-4c6a-bd63-d1adbe2e7631-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:48.083869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.083719 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs" pod "825917b2-9640-4c6a-bd63-d1adbe2e7631" (UID: "825917b2-9640-4c6a-bd63-d1adbe2e7631"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:48.083869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.083829 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6" pod "825917b2-9640-4c6a-bd63-d1adbe2e7631" (UID: "825917b2-9640-4c6a-bd63-d1adbe2e7631"). InnerVolumeSpecName "kube-api-access-zh5h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:48.085126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.085097 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls" pod "825917b2-9640-4c6a-bd63-d1adbe2e7631" (UID: "825917b2-9640-4c6a-bd63-d1adbe2e7631"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:48.184220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.182727 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:48.184220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.182760 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/825917b2-9640-4c6a-bd63-d1adbe2e7631-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:48.184220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.182774 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh5h6\" (UniqueName: \"kubernetes.io/projected/825917b2-9640-4c6a-bd63-d1adbe2e7631-kube-api-access-zh5h6\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:48.515702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.515663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:48.752553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.752417 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" event={"ID":"825917b2-9640-4c6a-bd63-d1adbe2e7631","Type":"ContainerDied","Data":"e8fab9733a8be32b774acca8ae09d7db3ae8a30326836eca8ca9e999b1d031f4"} Nov 03 16:52:48.752553 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.752526 4539 scope.go:122] "RemoveContainer" containerID="966796f5259ca16206af42011785039f4f9d2a9baa1398f8a5a8784e3224766e" Nov 03 16:52:48.752822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.752479 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l" Nov 03 16:52:48.799344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.799290 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l"] Nov 03 16:52:48.803969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:48.803933 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9160f5-746c597486-j4d9l"] Nov 03 16:52:49.554806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:49.554774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:49.740008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:49.739969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:50.338295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:50.338257 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:50.469244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:50.469213 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825917b2-9640-4c6a-bd63-d1adbe2e7631" path="/var/lib/kubelet/pods/825917b2-9640-4c6a-bd63-d1adbe2e7631/volumes" Nov 03 16:52:50.521522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:50.521489 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:50.692255 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:50.692170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:50.884973 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:50.884932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:51.354243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:51.354194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:51.702789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:51.702681 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf"] Nov 03 16:52:51.703315 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:51.702980 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" containerID="cri-o://23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34" gracePeriod=30 Nov 03 16:52:51.777440 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:51.777396 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:52.407405 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:52.407374 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:52.573667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:52.573635 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:52.896733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:52.896690 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:53.080937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:53.080899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:54.764836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:54.764804 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:56.157842 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:56.157804 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:52:56.531174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:56.531139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:56.824062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:56.824015 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:56.985611 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:56.985585 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:52:57.017153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017120 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd"] Nov 03 16:52:57.017896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017875 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ba167e1b-d5ea-4874-a11c-6a09b710bd8b" containerName="main" Nov 03 16:52:57.017979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017899 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba167e1b-d5ea-4874-a11c-6a09b710bd8b" containerName="main" Nov 03 16:52:57.017979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017938 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" Nov 03 16:52:57.017979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017945 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" Nov 03 16:52:57.017979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017957 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1a3e0dcd-871f-4076-bd65-4c6f99d95201" containerName="main" Nov 03 16:52:57.017979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017964 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3e0dcd-871f-4076-bd65-4c6f99d95201" containerName="main" Nov 03 16:52:57.017979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017976 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.017983 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018008 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a18d1788-acbf-4a39-8b9d-f59325436d16" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018015 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18d1788-acbf-4a39-8b9d-f59325436d16" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018024 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0641c0c8-4413-4dbc-a46e-8612fda4f05b" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018031 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0641c0c8-4413-4dbc-a46e-8612fda4f05b" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018040 4539 cpu_manager.go:414] "containerMap: removing container" podUID="825917b2-9640-4c6a-bd63-d1adbe2e7631" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018048 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825917b2-9640-4c6a-bd63-d1adbe2e7631" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018058 4539 cpu_manager.go:414] "containerMap: removing container" podUID="7653bca9-7e28-4343-beff-c04c7e9b8be3" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018065 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7653bca9-7e28-4343-beff-c04c7e9b8be3" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018076 4539 cpu_manager.go:414] "containerMap: removing container" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018083 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018100 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018108 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018120 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018127 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018143 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018149 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018160 4539 cpu_manager.go:414] "containerMap: removing container" podUID="eea3c944-079d-4f52-955b-bbd2f4d9e311" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018167 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea3c944-079d-4f52-955b-bbd2f4d9e311" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018178 4539 cpu_manager.go:414] "containerMap: removing container" podUID="8d4092e9-0202-4395-809c-fa9bad375cf3" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018185 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4092e9-0202-4395-809c-fa9bad375cf3" containerName="maas-api" Nov 03 16:52:57.018286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018296 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018308 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018317 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea3c944-079d-4f52-955b-bbd2f4d9e311" containerName="maas-api" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018326 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4092e9-0202-4395-809c-fa9bad375cf3" containerName="maas-api" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018336 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba167e1b-d5ea-4874-a11c-6a09b710bd8b" containerName="main" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018350 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6167316-2853-40c1-9357-1b10c4b471a5" containerName="istio-proxy" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018357 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3e0dcd-871f-4076-bd65-4c6f99d95201" containerName="main" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018365 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerName="istio-proxy" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018376 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18d1788-acbf-4a39-8b9d-f59325436d16" containerName="maas-api" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018382 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0641c0c8-4413-4dbc-a46e-8612fda4f05b" containerName="maas-api" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018390 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="825917b2-9640-4c6a-bd63-d1adbe2e7631" containerName="maas-api" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018398 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="7653bca9-7e28-4343-beff-c04c7e9b8be3" containerName="maas-api" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018406 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="72285235-afd8-4fec-87ea-88b01e1d3161" containerName="istio-proxy" Nov 03 16:52:57.019247 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.018415 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5586617a-e287-4fdd-9d22-ad436f7f385c" containerName="istio-proxy" Nov 03 16:52:57.022302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.022281 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.031845 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032055 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032110 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032148 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032215 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032217 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032265 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032300 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032409 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032416 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032514 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token\") pod \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\" (UID: \"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450\") " Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032721 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgdv\" (UniqueName: \"kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032866 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.032988 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.033014 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.033057 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.033087 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.033290 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:57.035303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.034158 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:57.039416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.039134 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:57.039740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.039569 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:52:57.039740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.039704 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:52:57.040264 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.040184 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:57.040674 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.040635 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:57.047808 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.047782 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd"] Nov 03 16:52:57.051469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.050510 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb" pod "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" (UID: "8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450"). InnerVolumeSpecName "kube-api-access-fnddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:57.134518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134391 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgdv\" (UniqueName: \"kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.134518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134510 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134589 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134673 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134690 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134702 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134713 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134727 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134735 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.134868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.134743 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnddb\" (UniqueName: \"kubernetes.io/projected/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450-kube-api-access-fnddb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:52:57.135347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.135299 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.137071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.137047 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.155721 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.155689 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgdv\" (UniqueName: \"kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv\") pod \"maas-api-revoke-sub-6da878e9-wp6xd\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.353591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.353560 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:52:57.656217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.656191 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd"] Nov 03 16:52:57.657822 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:52:57.657755 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e09a396_99f5_40a0_899c_cba468ab1655.slice/crio-f0d9579203dc83a0d912ac7c4a9bf082ca9653da479f588fb0c4c3c4507cbde0 WatchSource:0}: Error finding container f0d9579203dc83a0d912ac7c4a9bf082ca9653da479f588fb0c4c3c4507cbde0: Status 404 returned error can't find the container with id f0d9579203dc83a0d912ac7c4a9bf082ca9653da479f588fb0c4c3c4507cbde0 Nov 03 16:52:57.816739 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.816707 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:57.818592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.818546 4539 generic.go:356] "Generic (PLEG): container finished" podID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" containerID="23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34" exitCode=0 Nov 03 16:52:57.818828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.818811 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" Nov 03 16:52:57.819775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.819749 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" event={"ID":"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450","Type":"ContainerDied","Data":"23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34"} Nov 03 16:52:57.819914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.819810 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf" event={"ID":"8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450","Type":"ContainerDied","Data":"34a76c180e4e200c3cef4dfb553cf70923011be55186209febaa2b1892bacdb3"} Nov 03 16:52:57.819914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.819843 4539 scope.go:122] "RemoveContainer" containerID="23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34" Nov 03 16:52:57.846593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.846559 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" event={"ID":"4e09a396-99f5-40a0-899c-cba468ab1655","Type":"ContainerStarted","Data":"f0d9579203dc83a0d912ac7c4a9bf082ca9653da479f588fb0c4c3c4507cbde0"} Nov 03 16:52:57.855813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.855791 4539 scope.go:122] "RemoveContainer" containerID="962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea" Nov 03 16:52:57.865522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.865495 4539 scope.go:122] "RemoveContainer" containerID="23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34" Nov 03 16:52:57.865838 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:57.865819 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34\": container with ID starting with 23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34 not found: ID does not exist" containerID="23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34" Nov 03 16:52:57.865878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.865849 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34"} err="failed to get container status \"23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34\": rpc error: code = NotFound desc = could not find container \"23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34\": container with ID starting with 23b2f089c4748d9641b19a608491d6a04a68731856887d3a911143e1da461a34 not found: ID does not exist" Nov 03 16:52:57.865878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.865868 4539 scope.go:122] "RemoveContainer" containerID="962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea" Nov 03 16:52:57.866148 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:52:57.866125 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea\": container with ID starting with 962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea not found: ID does not exist" containerID="962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea" Nov 03 16:52:57.866282 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.866153 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea"} err="failed to get container status \"962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea\": rpc error: code = NotFound desc = could not find container \"962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea\": container with ID starting with 962ca03d38c4c0d7e6ebf6493346e9a7bb4f7fda9971a3eee33df76f5dbb1fea not found: ID does not exist" Nov 03 16:52:57.888975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.888943 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf"] Nov 03 16:52:57.899401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:57.899371 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9160f5-openshift-default-7b7cbb6965-z9mdf"] Nov 03 16:52:58.225521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:58.222083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:58.397700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:58.397670 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:58.474451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:58.473412 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450" path="/var/lib/kubelet/pods/8c00b5a6-8e87-44a1-bd0f-eea4b2dd7450/volumes" Nov 03 16:52:58.560587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:58.558995 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:52:58.855916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:58.855827 4539 generic.go:356] "Generic (PLEG): container finished" podID="4e09a396-99f5-40a0-899c-cba468ab1655" containerID="a7cf16bbaf8ff85f6f70b0cca5d78809984fe217f4b4d2557ec317060b26ce11" exitCode=0 Nov 03 16:52:58.856084 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:52:58.855921 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" event={"ID":"4e09a396-99f5-40a0-899c-cba468ab1655","Type":"ContainerDied","Data":"a7cf16bbaf8ff85f6f70b0cca5d78809984fe217f4b4d2557ec317060b26ce11"} Nov 03 16:53:00.029505 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.029477 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:53:00.076544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.076508 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca\") pod \"4e09a396-99f5-40a0-899c-cba468ab1655\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " Nov 03 16:53:00.076775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.076579 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv\" (UniqueName: \"kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv\") pod \"4e09a396-99f5-40a0-899c-cba468ab1655\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " Nov 03 16:53:00.076775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.076717 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca\") pod \"4e09a396-99f5-40a0-899c-cba468ab1655\" (UID: \"4e09a396-99f5-40a0-899c-cba468ab1655\") " Nov 03 16:53:00.076981 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.076864 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca" pod "4e09a396-99f5-40a0-899c-cba468ab1655" (UID: "4e09a396-99f5-40a0-899c-cba468ab1655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:00.077227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.077202 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:00.080660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.080485 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv" pod "4e09a396-99f5-40a0-899c-cba468ab1655" (UID: "4e09a396-99f5-40a0-899c-cba468ab1655"). InnerVolumeSpecName "kube-api-access-lkgdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:00.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.082538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:00.083864 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.083834 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca" pod "4e09a396-99f5-40a0-899c-cba468ab1655" (UID: "4e09a396-99f5-40a0-899c-cba468ab1655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:00.178651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.178562 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgdv\" (UniqueName: \"kubernetes.io/projected/4e09a396-99f5-40a0-899c-cba468ab1655-kube-api-access-lkgdv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:00.178651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.178596 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e09a396-99f5-40a0-899c-cba468ab1655-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:00.871850 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.871817 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" Nov 03 16:53:00.871850 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.871825 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd" event={"ID":"4e09a396-99f5-40a0-899c-cba468ab1655","Type":"ContainerDied","Data":"f0d9579203dc83a0d912ac7c4a9bf082ca9653da479f588fb0c4c3c4507cbde0"} Nov 03 16:53:00.872073 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:00.871873 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d9579203dc83a0d912ac7c4a9bf082ca9653da479f588fb0c4c3c4507cbde0" Nov 03 16:53:01.737458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:01.735236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:03.286750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:03.286706 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:04.987811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:04.987763 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:06.544662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:06.544630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:06.695510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:06.695478 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:06.853169 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:06.853077 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:07.045112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.045068 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9"] Nov 03 16:53:07.045370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.045343 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" podUID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" containerName="discovery" containerID="cri-o://1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a" gracePeriod=30 Nov 03 16:53:07.344268 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.344237 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:53:07.437436 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.437317 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m"] Nov 03 16:53:07.438183 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.438158 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4e09a396-99f5-40a0-899c-cba468ab1655" containerName="revoke-keys" Nov 03 16:53:07.438329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.438187 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e09a396-99f5-40a0-899c-cba468ab1655" containerName="revoke-keys" Nov 03 16:53:07.438329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.438207 4539 cpu_manager.go:414] "containerMap: removing container" podUID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" containerName="discovery" Nov 03 16:53:07.438329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.438216 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" containerName="discovery" Nov 03 16:53:07.438546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.438410 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e09a396-99f5-40a0-899c-cba468ab1655" containerName="revoke-keys" Nov 03 16:53:07.438546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.438507 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" containerName="discovery" Nov 03 16:53:07.442905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.442630 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.459818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.456672 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:07.464625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464173 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m"] Nov 03 16:53:07.464625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464618 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.464888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464688 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.464888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464726 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.464888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464787 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.464888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464880 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.465138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.464952 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.465138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.465027 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap\") pod \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\" (UID: \"3292ce99-8bc8-4ece-91df-7dcb1d931c5e\") " Nov 03 16:53:07.465938 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.465889 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:07.466085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.465973 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:07.477787 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.477731 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:07.491040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.490984 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "kube-api-access-vg6vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:07.499962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.499935 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:07.500187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.500062 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:07.500873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.500580 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:07.503691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.503632 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig" pod "3292ce99-8bc8-4ece-91df-7dcb1d931c5e" (UID: "3292ce99-8bc8-4ece-91df-7dcb1d931c5e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:07.566895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.566854 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.567377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567056 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2674\" (UniqueName: \"kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.567377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567182 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.567377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567348 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.567377 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567367 4539 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-cacerts\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.567601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567384 4539 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-ca-configmap\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.567601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567401 4539 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-kubeconfig\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.567601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567416 4539 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-local-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.567601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567448 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6vg\" (UniqueName: \"kubernetes.io/projected/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-kube-api-access-vg6vg\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.567601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.567462 4539 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3292ce99-8bc8-4ece-91df-7dcb1d931c5e-istio-csr-dns-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:07.668952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.668920 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.669137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.669005 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.669137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.669100 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2674\" (UniqueName: \"kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.669794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.669752 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.672120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.672084 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.691303 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.691234 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2674\" (UniqueName: \"kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674\") pod \"maas-api-revoke-sub-6d78bebd-jv48m\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.769711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.769672 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:07.922058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.922017 4539 generic.go:356] "Generic (PLEG): container finished" podID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" containerID="1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a" exitCode=0 Nov 03 16:53:07.922254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.922074 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" Nov 03 16:53:07.922254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.922132 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" event={"ID":"3292ce99-8bc8-4ece-91df-7dcb1d931c5e","Type":"ContainerDied","Data":"1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a"} Nov 03 16:53:07.922254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.922175 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9" event={"ID":"3292ce99-8bc8-4ece-91df-7dcb1d931c5e","Type":"ContainerDied","Data":"b7306908e59c5178661b4b1873fa1fe085fe340d7d1d996dbefcf871003069b6"} Nov 03 16:53:07.922254 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.922197 4539 scope.go:122] "RemoveContainer" containerID="1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a" Nov 03 16:53:07.934197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.934147 4539 scope.go:122] "RemoveContainer" containerID="1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a" Nov 03 16:53:07.934589 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:53:07.934554 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a\": container with ID starting with 1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a not found: ID does not exist" containerID="1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a" Nov 03 16:53:07.934760 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.934600 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a"} err="failed to get container status \"1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a\": rpc error: code = NotFound desc = could not find container \"1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a\": container with ID starting with 1c9c808accd93c10f547a03654a5f129182d23caa3108e23d2d7a0b4355d969a not found: ID does not exist" Nov 03 16:53:07.972476 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.972370 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9"] Nov 03 16:53:07.977139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:07.977107 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k6pb9"] Nov 03 16:53:08.073975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:08.073763 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m"] Nov 03 16:53:08.076402 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:08.076368 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1514118_889d_479c_ad17_9a7f322e086e.slice/crio-cac458b3869968b4f5fdc239671801c25532d8084141143bfa2789eebcc27365 WatchSource:0}: Error finding container cac458b3869968b4f5fdc239671801c25532d8084141143bfa2789eebcc27365: Status 404 returned error can't find the container with id cac458b3869968b4f5fdc239671801c25532d8084141143bfa2789eebcc27365 Nov 03 16:53:08.306370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:08.306341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:08.470339 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:08.470300 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3292ce99-8bc8-4ece-91df-7dcb1d931c5e" path="/var/lib/kubelet/pods/3292ce99-8bc8-4ece-91df-7dcb1d931c5e/volumes" Nov 03 16:53:08.939006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:08.938903 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" event={"ID":"b1514118-889d-479c-ad17-9a7f322e086e","Type":"ContainerStarted","Data":"cbfcfba032e3eb5906111802dcdbaa2952503d8bdf2cecdad20d1213f5424d38"} Nov 03 16:53:08.939006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:08.938976 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" event={"ID":"b1514118-889d-479c-ad17-9a7f322e086e","Type":"ContainerStarted","Data":"cac458b3869968b4f5fdc239671801c25532d8084141143bfa2789eebcc27365"} Nov 03 16:53:08.988835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:08.988776 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" podStartSLOduration=1.988761019 podStartE2EDuration="1.988761019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:08.985672114 +0000 UTC m=+2332.678943147" watchObservedRunningTime="2026-11-03 16:53:08.988761019 +0000 UTC m=+2332.682032040" Nov 03 16:53:09.949075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:09.949044 4539 generic.go:356] "Generic (PLEG): container finished" podID="b1514118-889d-479c-ad17-9a7f322e086e" containerID="cbfcfba032e3eb5906111802dcdbaa2952503d8bdf2cecdad20d1213f5424d38" exitCode=0 Nov 03 16:53:09.949514 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:09.949095 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" event={"ID":"b1514118-889d-479c-ad17-9a7f322e086e","Type":"ContainerDied","Data":"cbfcfba032e3eb5906111802dcdbaa2952503d8bdf2cecdad20d1213f5424d38"} Nov 03 16:53:09.976050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:09.976014 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:11.110103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.110071 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:11.211307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.211218 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674\" (UniqueName: \"kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674\") pod \"b1514118-889d-479c-ad17-9a7f322e086e\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " Nov 03 16:53:11.211500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.211466 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca\") pod \"b1514118-889d-479c-ad17-9a7f322e086e\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " Nov 03 16:53:11.211573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.211519 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca\") pod \"b1514118-889d-479c-ad17-9a7f322e086e\" (UID: \"b1514118-889d-479c-ad17-9a7f322e086e\") " Nov 03 16:53:11.212064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.211996 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca" pod "b1514118-889d-479c-ad17-9a7f322e086e" (UID: "b1514118-889d-479c-ad17-9a7f322e086e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:11.213869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.213840 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674" pod "b1514118-889d-479c-ad17-9a7f322e086e" (UID: "b1514118-889d-479c-ad17-9a7f322e086e"). InnerVolumeSpecName "kube-api-access-b2674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:11.215069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.215049 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca" pod "b1514118-889d-479c-ad17-9a7f322e086e" (UID: "b1514118-889d-479c-ad17-9a7f322e086e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:11.313544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.313512 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:11.313544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.313541 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1514118-889d-479c-ad17-9a7f322e086e-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:11.313544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.313551 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2674\" (UniqueName: \"kubernetes.io/projected/b1514118-889d-479c-ad17-9a7f322e086e-kube-api-access-b2674\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:11.704497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.704455 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:11.962332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.962246 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" event={"ID":"b1514118-889d-479c-ad17-9a7f322e086e","Type":"ContainerDied","Data":"cac458b3869968b4f5fdc239671801c25532d8084141143bfa2789eebcc27365"} Nov 03 16:53:11.962332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.962281 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m" Nov 03 16:53:11.962332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.962296 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac458b3869968b4f5fdc239671801c25532d8084141143bfa2789eebcc27365" Nov 03 16:53:11.993681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:11.993651 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:13.477243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:13.477195 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:15.212191 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:15.212114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:17.441452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:17.432096 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:18.979848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:18.979816 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:18.992406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:18.992376 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs"] Nov 03 16:53:18.992639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:18.992609 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" containerID="cri-o://8e43281e796d196e520beafc386170056ced47ff95dbe5a2d66a7fadc2a37062" gracePeriod=30 Nov 03 16:53:19.664776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.664739 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj"] Nov 03 16:53:19.665222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.665209 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b1514118-889d-479c-ad17-9a7f322e086e" containerName="revoke-keys" Nov 03 16:53:19.665272 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.665224 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1514118-889d-479c-ad17-9a7f322e086e" containerName="revoke-keys" Nov 03 16:53:19.665347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.665325 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1514118-889d-479c-ad17-9a7f322e086e" containerName="revoke-keys" Nov 03 16:53:19.670070 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.670047 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.687477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.687442 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24"] Nov 03 16:53:19.692045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.692013 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.706768 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.706737 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj"] Nov 03 16:53:19.714402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.714373 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6"] Nov 03 16:53:19.718826 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.718800 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.734125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.734103 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24"] Nov 03 16:53:19.749219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.749188 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6"] Nov 03 16:53:19.763540 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.763493 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.763736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.763625 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.763736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.763668 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmlv\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.763736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.763696 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.763914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.763862 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.763914 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.763907 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.764030 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.764000 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.764083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.764045 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.764137 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.764117 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.864895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.864858 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.864905 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.864935 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmlv\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.864963 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.864997 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97tt\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865125 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.865215 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865204 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865288 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865315 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865343 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865416 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865398 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865459 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.865526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865504 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.865562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865535 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865573 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865649 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865630 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.865682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865669 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865715 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865766 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865745 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.865798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865767 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.865798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865773 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.865911 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865896 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6fw\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.865979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.865962 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.866023 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866002 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.866066 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866036 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.866104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866067 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.866257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866238 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.866311 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866296 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.866383 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866368 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.866419 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866399 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.866614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866598 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.866655 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.866600 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.868038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.868017 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.868677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.868659 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.886999 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.886971 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.887131 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.887033 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmlv\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.968079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.967991 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968054 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968079 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968076 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968104 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968121 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6fw\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968151 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968203 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968227 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968297 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968326 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968378 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968465 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97tt\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968530 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968557 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968588 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968635 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968655 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968673 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968717 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968885 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.968974 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968906 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.969414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.968981 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.969414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.969139 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.969414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.969161 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.969414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.969182 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.969414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.969240 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.969642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.969487 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.970182 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.970157 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.970814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.970789 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.971597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.971575 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.971673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.971621 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.971673 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.971628 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.984851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.984823 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:19.990849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.990814 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6fw\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.994233 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.994210 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:19.994500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.994480 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:19.994738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:19.994718 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97tt\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:20.018621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.018584 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:20.048651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.048606 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:20.325057 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:20.325016 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58cd0726_b63e_432f_818a_0e089054a8e3.slice/crio-2a2150d584f995cec388a25f441ee62e5e0a9cbfd1f338a34fb17bbfe44f7cf3 WatchSource:0}: Error finding container 2a2150d584f995cec388a25f441ee62e5e0a9cbfd1f338a34fb17bbfe44f7cf3: Status 404 returned error can't find the container with id 2a2150d584f995cec388a25f441ee62e5e0a9cbfd1f338a34fb17bbfe44f7cf3 Nov 03 16:53:20.325193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.325167 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj"] Nov 03 16:53:20.327628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.327598 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.327743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.327670 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.327743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.327704 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.352610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.352585 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24"] Nov 03 16:53:20.363725 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:20.363693 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9394bf9b_9771_4f6e_97cc_e63802c79795.slice/crio-b0e520181c67db0b99fabac49cd0a5c791d32fe6ab00f37926955ac5b3781f81 WatchSource:0}: Error finding container b0e520181c67db0b99fabac49cd0a5c791d32fe6ab00f37926955ac5b3781f81: Status 404 returned error can't find the container with id b0e520181c67db0b99fabac49cd0a5c791d32fe6ab00f37926955ac5b3781f81 Nov 03 16:53:20.366611 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.366577 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.366713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.366653 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.366713 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.366683 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.399749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.399689 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6"] Nov 03 16:53:20.531662 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:20.531610 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d57cdfd_557b_4f81_b661_7b87125e8529.slice/crio-f5bf0f826219971159ed53a3262e71e19b6489b6532a57dd3a6ddb4c6ca588ff WatchSource:0}: Error finding container f5bf0f826219971159ed53a3262e71e19b6489b6532a57dd3a6ddb4c6ca588ff: Status 404 returned error can't find the container with id f5bf0f826219971159ed53a3262e71e19b6489b6532a57dd3a6ddb4c6ca588ff Nov 03 16:53:20.537396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.535104 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.537396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.535190 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.537396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.535233 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:20.699125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:20.699072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:21.157886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.157850 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" event={"ID":"9394bf9b-9771-4f6e-97cc-e63802c79795","Type":"ContainerStarted","Data":"67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda"} Nov 03 16:53:21.158379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.157913 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" event={"ID":"9394bf9b-9771-4f6e-97cc-e63802c79795","Type":"ContainerStarted","Data":"b0e520181c67db0b99fabac49cd0a5c791d32fe6ab00f37926955ac5b3781f81"} Nov 03 16:53:21.159495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.159465 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" event={"ID":"58cd0726-b63e-432f-818a-0e089054a8e3","Type":"ContainerStarted","Data":"404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0"} Nov 03 16:53:21.159663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.159519 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" event={"ID":"58cd0726-b63e-432f-818a-0e089054a8e3","Type":"ContainerStarted","Data":"2a2150d584f995cec388a25f441ee62e5e0a9cbfd1f338a34fb17bbfe44f7cf3"} Nov 03 16:53:21.161135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.161110 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" event={"ID":"6d57cdfd-557b-4f81-b661-7b87125e8529","Type":"ContainerStarted","Data":"05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e"} Nov 03 16:53:21.161238 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.161156 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" event={"ID":"6d57cdfd-557b-4f81-b661-7b87125e8529","Type":"ContainerStarted","Data":"f5bf0f826219971159ed53a3262e71e19b6489b6532a57dd3a6ddb4c6ca588ff"} Nov 03 16:53:21.234549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.234486 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podStartSLOduration=2.234460446 podStartE2EDuration="2.234460446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:21.220326525 +0000 UTC m=+2344.913597546" watchObservedRunningTime="2026-11-03 16:53:21.234460446 +0000 UTC m=+2344.927731466" Nov 03 16:53:21.266056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.265981 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podStartSLOduration=2.265957932 podStartE2EDuration="2.265957932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:21.262411417 +0000 UTC m=+2344.955682441" watchObservedRunningTime="2026-11-03 16:53:21.265957932 +0000 UTC m=+2344.959228955" Nov 03 16:53:21.314096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.314011 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podStartSLOduration=2.3139873 podStartE2EDuration="2.3139873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:21.307480646 +0000 UTC m=+2345.000751665" watchObservedRunningTime="2026-11-03 16:53:21.3139873 +0000 UTC m=+2345.007258323" Nov 03 16:53:21.986621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.985724 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:21.987780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.987745 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:21.987948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:21.987819 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:22.020934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.020892 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:22.024634 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.024584 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:22.024962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.024915 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:22.063448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.062808 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:22.067453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.066569 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:22.067453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.066629 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:22.714449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.711020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:22.897450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.897178 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:22.986522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.986393 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:22.986797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:22.986767 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:23.025038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.024989 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:23.025369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.025321 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:23.055076 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.055024 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:23.055418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.055370 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:23.167360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.164833 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7"] Nov 03 16:53:23.177465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.170093 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.186496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.182161 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7"] Nov 03 16:53:23.189890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.189690 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:23.246226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.246141 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:23.339449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.318472 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.339449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.318617 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptztx\" (UniqueName: \"kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.339449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.318722 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.440450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.433163 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:23.498504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.479031 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.498504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.479195 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptztx\" (UniqueName: \"kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.498504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.479320 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.498504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.480927 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.499321 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.499287 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl"] Nov 03 16:53:23.510460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.505838 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.548453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.521850 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a11d94-openshift-default-dockercfg-lqlpz\"" Nov 03 16:53:23.548453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.523369 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.548453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.530800 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl"] Nov 03 16:53:23.644489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.605716 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptztx\" (UniqueName: \"kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx\") pod \"maas-api-revoke-sub-fb1be90c-p8xn7\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.695450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.695194 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.695450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.695272 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.695450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.695342 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.695450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.695371 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.695450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.695400 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.813452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.799035 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.813452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.799138 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx547\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.813452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.799168 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.813452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.799217 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.908453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.907480 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:23.908453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.908415 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.908733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.908492 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.908733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.908553 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.908733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.908623 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.908733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.908651 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.908733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.908678 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.909487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.909460 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.930210 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.910216 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.930365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.910475 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.930496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.915731 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.930715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.930695 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx547\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.931230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.930137 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.931375 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.928253 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.931533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.931516 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.963453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.961785 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.996225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.982051 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx547\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:23.996225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.990896 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:23.996225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:23.990967 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:24.011507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.005135 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:24.019639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.019590 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:24.027629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.019921 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:24.027959 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.021828 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:24.028344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.028308 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo\") pod \"e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:24.056603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.056564 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 03 16:53:24.063449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.056806 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 03 16:53:24.067454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.067058 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:24.067454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.067171 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:24.091178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.091076 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:24.267718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.266491 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:24.329993 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.328527 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:24.538899 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.529530 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:24.538899 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.538784 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:24.623343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.623250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:24.648409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.647927 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:24.648409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.648340 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 16:53:24.992730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.992238 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:24.992730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:24.992300 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:25.057880 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.057845 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7"] Nov 03 16:53:25.058207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.058184 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:25.058381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.058356 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:25.058747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.058719 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:25.058839 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.058763 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:25.104582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.103697 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:25.274460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.272635 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:25.274460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.273494 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:53:25.388817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.388786 4539 generic.go:356] "Generic (PLEG): container finished" podID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerID="8e43281e796d196e520beafc386170056ced47ff95dbe5a2d66a7fadc2a37062" exitCode=0 Nov 03 16:53:25.389112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.389091 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" event={"ID":"3743f418-0f1d-46b4-b0db-8f7a50bf4942","Type":"ContainerDied","Data":"8e43281e796d196e520beafc386170056ced47ff95dbe5a2d66a7fadc2a37062"} Nov 03 16:53:25.391525 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.391497 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" event={"ID":"d4d21824-fbd7-4524-93a9-1275670dc513","Type":"ContainerStarted","Data":"a331ff168d8c983c65579c5cee19dfab35383813d24be1abe0a169ffe96b5072"} Nov 03 16:53:25.398964 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.398925 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl"] Nov 03 16:53:25.426179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.413737 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:53:25.694642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.693330 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:25.694642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.693445 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:25.694642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.693492 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:25.845881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.845844 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.954791 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.954850 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.954919 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.954989 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.955042 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.955185 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.955215 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.955299 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.955343 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo\") pod \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\" (UID: \"3743f418-0f1d-46b4-b0db-8f7a50bf4942\") " Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.970660 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:25.975450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.970779 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:25.984717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.976321 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:25.984717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.976413 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:25.984717 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.981527 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.985472 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.985501 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.985515 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.985528 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.985540 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.987402 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:25.989888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.987470 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:26.006625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:25.997952 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:53:26.023224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.021095 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:26.023224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.021147 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:26.050589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.050463 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:26.050589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.050523 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:26.088397 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.088327 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:26.195997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.195953 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:26.221698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.221630 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "kube-api-access-vgsrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:26.255746 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.255676 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy" pod "3743f418-0f1d-46b4-b0db-8f7a50bf4942" (UID: "3743f418-0f1d-46b4-b0db-8f7a50bf4942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:26.294260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.293349 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsrv\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-kube-api-access-vgsrv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:26.294260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.293384 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:26.294260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.293399 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3743f418-0f1d-46b4-b0db-8f7a50bf4942-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:26.431488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.426988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:26.492809 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:53:26.486878 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:53:26.510062 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.505082 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" Nov 03 16:53:26.981458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.981391 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs" event={"ID":"3743f418-0f1d-46b4-b0db-8f7a50bf4942","Type":"ContainerDied","Data":"e44d8c1b680b1bf11a1cf17bcea1da918b58394de0f7113244beeb030c125feb"} Nov 03 16:53:26.982820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.981681 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" event={"ID":"d4d21824-fbd7-4524-93a9-1275670dc513","Type":"ContainerStarted","Data":"63a8be7fe7519c4e6b9eaaec88c97e30f68155cbd042c790525961da093f22ba"} Nov 03 16:53:26.982820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.981721 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" event={"ID":"c50b512a-6cf5-40be-9d89-893bd82072ad","Type":"ContainerStarted","Data":"eb050f108dae63b2d159448e5adbe508165ae422a9f7a7564b282cbb1c22449d"} Nov 03 16:53:26.982820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.982653 4539 scope.go:122] "RemoveContainer" containerID="8e43281e796d196e520beafc386170056ced47ff95dbe5a2d66a7fadc2a37062" Nov 03 16:53:26.997533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.997239 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:26.997533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:26.997300 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:27.027452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.027391 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:27.094573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.094533 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:27.095160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.071209 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs"] Nov 03 16:53:27.095344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.095332 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:27.095503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.071170 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:27.095631 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.095610 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:27.095735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.092976 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:53:27.095835 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.095819 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:53:27.172465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.168648 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kngcs"] Nov 03 16:53:27.203841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.202007 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" podStartSLOduration=4.201983279 podStartE2EDuration="4.201983279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:27.093516197 +0000 UTC m=+2350.786787220" watchObservedRunningTime="2026-11-03 16:53:27.201983279 +0000 UTC m=+2350.895254303" Nov 03 16:53:27.228512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.227134 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podStartSLOduration=4.227109271 podStartE2EDuration="4.227109271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:27.156045894 +0000 UTC m=+2350.849316931" watchObservedRunningTime="2026-11-03 16:53:27.227109271 +0000 UTC m=+2350.920380294" Nov 03 16:53:27.640148 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.640108 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" event={"ID":"c50b512a-6cf5-40be-9d89-893bd82072ad","Type":"ContainerStarted","Data":"08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45"} Nov 03 16:53:27.652287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.652040 4539 generic.go:356] "Generic (PLEG): container finished" podID="d4d21824-fbd7-4524-93a9-1275670dc513" containerID="63a8be7fe7519c4e6b9eaaec88c97e30f68155cbd042c790525961da093f22ba" exitCode=0 Nov 03 16:53:27.652287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.652156 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" event={"ID":"d4d21824-fbd7-4524-93a9-1275670dc513","Type":"ContainerDied","Data":"63a8be7fe7519c4e6b9eaaec88c97e30f68155cbd042c790525961da093f22ba"} Nov 03 16:53:27.986058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.985967 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 16:53:27.986058 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:27.986036 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 16:53:28.021857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.020264 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:28.021857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.020328 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:28.029881 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.022999 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:28.061591 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.061550 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 16:53:28.061892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.061860 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 16:53:28.100447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.096508 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:28.475301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.475261 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" path="/var/lib/kubelet/pods/3743f418-0f1d-46b4-b0db-8f7a50bf4942/volumes" Nov 03 16:53:28.662406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.662006 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:28.663468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.663409 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:53:28.934814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.934772 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:28.992940 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.992897 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca\") pod \"d4d21824-fbd7-4524-93a9-1275670dc513\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " Nov 03 16:53:28.993142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.993020 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca\") pod \"d4d21824-fbd7-4524-93a9-1275670dc513\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " Nov 03 16:53:28.993142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.993099 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx\" (UniqueName: \"kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx\") pod \"d4d21824-fbd7-4524-93a9-1275670dc513\" (UID: \"d4d21824-fbd7-4524-93a9-1275670dc513\") " Nov 03 16:53:28.995619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:28.994834 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca" pod "d4d21824-fbd7-4524-93a9-1275670dc513" (UID: "d4d21824-fbd7-4524-93a9-1275670dc513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:29.002052 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.001588 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca" pod "d4d21824-fbd7-4524-93a9-1275670dc513" (UID: "d4d21824-fbd7-4524-93a9-1275670dc513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:29.005602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.003970 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx" pod "d4d21824-fbd7-4524-93a9-1275670dc513" (UID: "d4d21824-fbd7-4524-93a9-1275670dc513"). InnerVolumeSpecName "kube-api-access-ptztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:29.021493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.019468 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:53:29.021493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.019529 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 16:53:29.095197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.095157 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:29.095889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.095807 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d21824-fbd7-4524-93a9-1275670dc513-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:29.095889 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.095838 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptztx\" (UniqueName: \"kubernetes.io/projected/d4d21824-fbd7-4524-93a9-1275670dc513-kube-api-access-ptztx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:29.567570 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.565312 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:29.702371 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.691014 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" Nov 03 16:53:29.722817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.721051 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7" event={"ID":"d4d21824-fbd7-4524-93a9-1275670dc513","Type":"ContainerDied","Data":"a331ff168d8c983c65579c5cee19dfab35383813d24be1abe0a169ffe96b5072"} Nov 03 16:53:29.722817 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.721275 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a331ff168d8c983c65579c5cee19dfab35383813d24be1abe0a169ffe96b5072" Nov 03 16:53:29.732237 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.732170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:29.851871 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.851776 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:29.853114 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.853093 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:29.854683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.854663 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:53:29.909099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.909046 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:29.910166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.910112 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:29.911554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:29.911532 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:53:30.316301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:30.316264 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:30.316821 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:30.316799 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:30.317874 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:30.317848 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:53:30.875701 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:30.874288 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:31.276822 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:31.276792 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:31.435948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:31.435918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:33.071111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:33.071078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:34.728618 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:34.728587 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:36.554238 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:36.554202 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:38.505558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:38.505509 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.070749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.070710 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:39.246587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.246545 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:53:39.246901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.246869 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.305338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.305287 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.417898 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.417789 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.426447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.423473 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.484329 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.484273 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:53:39.691025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:39.690653 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:40.204204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204166 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq"] Nov 03 16:53:40.204780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204760 4539 cpu_manager.go:414] "containerMap: removing container" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" Nov 03 16:53:40.204780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204778 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" Nov 03 16:53:40.204943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204802 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d4d21824-fbd7-4524-93a9-1275670dc513" containerName="revoke-keys" Nov 03 16:53:40.204943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204807 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d21824-fbd7-4524-93a9-1275670dc513" containerName="revoke-keys" Nov 03 16:53:40.204943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204886 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="3743f418-0f1d-46b4-b0db-8f7a50bf4942" containerName="istio-proxy" Nov 03 16:53:40.204943 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.204897 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d21824-fbd7-4524-93a9-1275670dc513" containerName="revoke-keys" Nov 03 16:53:40.208732 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.208710 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.229895 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.229865 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq"] Nov 03 16:53:40.234630 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.234567 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:40.234785 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.234644 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:40.357179 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.357145 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.357385 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.357315 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.357475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.357395 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pkq\" (UniqueName: \"kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.458486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.458377 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pkq\" (UniqueName: \"kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.458660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.458547 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.458660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.458634 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.459325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.459305 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.461370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.461341 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.480289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.480263 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pkq\" (UniqueName: \"kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq\") pod \"maas-api-revoke-sub-5ca14a94-4z5pq\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.524800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.524768 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:40.689563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.689524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:40.833135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.833106 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq"] Nov 03 16:53:40.835923 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:40.835887 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9c1086_d742_4cce_b02a_ed514d30cb72.slice/crio-415d4e455426bc22d9d1c1aa3691229bd35f0b62dc35af4f152dda0bf6285d77 WatchSource:0}: Error finding container 415d4e455426bc22d9d1c1aa3691229bd35f0b62dc35af4f152dda0bf6285d77: Status 404 returned error can't find the container with id 415d4e455426bc22d9d1c1aa3691229bd35f0b62dc35af4f152dda0bf6285d77 Nov 03 16:53:40.873365 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:40.873332 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" event={"ID":"5c9c1086-d742-4cce-b02a-ed514d30cb72","Type":"ContainerStarted","Data":"415d4e455426bc22d9d1c1aa3691229bd35f0b62dc35af4f152dda0bf6285d77"} Nov 03 16:53:41.881547 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:41.881516 4539 generic.go:356] "Generic (PLEG): container finished" podID="5c9c1086-d742-4cce-b02a-ed514d30cb72" containerID="7807da781bdbd1d8a2572b559ddb2f84b5b692171a27ef60fec3f4d487fc1373" exitCode=0 Nov 03 16:53:41.881997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:41.881601 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" event={"ID":"5c9c1086-d742-4cce-b02a-ed514d30cb72","Type":"ContainerDied","Data":"7807da781bdbd1d8a2572b559ddb2f84b5b692171a27ef60fec3f4d487fc1373"} Nov 03 16:53:42.245522 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.245413 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k"] Nov 03 16:53:42.249534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.249511 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.262468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.262443 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a11d94-serving-cert\"" Nov 03 16:53:42.276969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.276938 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k"] Nov 03 16:53:42.380500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.380465 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxqb\" (UniqueName: \"kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.380683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.380597 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.380683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.380630 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.380849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.380830 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.401794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.401762 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:42.481754 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.481718 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.481955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.481764 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.481955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.481945 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.482081 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.482006 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxqb\" (UniqueName: \"kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.482582 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.482550 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.484450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.484406 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.484551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.484466 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.504285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.504204 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxqb\" (UniqueName: \"kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb\") pod \"maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.562503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.562467 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:42.566020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.565984 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:42.890138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:42.890113 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k"] Nov 03 16:53:42.891546 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:42.891523 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601584a0_5ff2_4aa6_81bd_95c8115cac17.slice/crio-203dca0ae0d42b0ab34025a5b491098756b5c559a0d6891386d10ea33e046d11 WatchSource:0}: Error finding container 203dca0ae0d42b0ab34025a5b491098756b5c559a0d6891386d10ea33e046d11: Status 404 returned error can't find the container with id 203dca0ae0d42b0ab34025a5b491098756b5c559a0d6891386d10ea33e046d11 Nov 03 16:53:43.053006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.052975 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:43.193053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.192815 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq\" (UniqueName: \"kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq\") pod \"5c9c1086-d742-4cce-b02a-ed514d30cb72\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " Nov 03 16:53:43.193053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.192892 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca\") pod \"5c9c1086-d742-4cce-b02a-ed514d30cb72\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " Nov 03 16:53:43.193053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.192926 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca\") pod \"5c9c1086-d742-4cce-b02a-ed514d30cb72\" (UID: \"5c9c1086-d742-4cce-b02a-ed514d30cb72\") " Nov 03 16:53:43.193765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.193735 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca" pod "5c9c1086-d742-4cce-b02a-ed514d30cb72" (UID: "5c9c1086-d742-4cce-b02a-ed514d30cb72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:43.195193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.195163 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq" pod "5c9c1086-d742-4cce-b02a-ed514d30cb72" (UID: "5c9c1086-d742-4cce-b02a-ed514d30cb72"). InnerVolumeSpecName "kube-api-access-55pkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:43.195917 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.195885 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca" pod "5c9c1086-d742-4cce-b02a-ed514d30cb72" (UID: "5c9c1086-d742-4cce-b02a-ed514d30cb72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:43.294159 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.294124 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pkq\" (UniqueName: \"kubernetes.io/projected/5c9c1086-d742-4cce-b02a-ed514d30cb72-kube-api-access-55pkq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:43.294159 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.294156 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:43.294159 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.294166 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c9c1086-d742-4cce-b02a-ed514d30cb72-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:53:43.901171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.900836 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" event={"ID":"601584a0-5ff2-4aa6-81bd-95c8115cac17","Type":"ContainerStarted","Data":"d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787"} Nov 03 16:53:43.901171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.900909 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" event={"ID":"601584a0-5ff2-4aa6-81bd-95c8115cac17","Type":"ContainerStarted","Data":"203dca0ae0d42b0ab34025a5b491098756b5c559a0d6891386d10ea33e046d11"} Nov 03 16:53:43.901171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.901111 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:43.904147 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.904111 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" event={"ID":"5c9c1086-d742-4cce-b02a-ed514d30cb72","Type":"ContainerDied","Data":"415d4e455426bc22d9d1c1aa3691229bd35f0b62dc35af4f152dda0bf6285d77"} Nov 03 16:53:43.904313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.904164 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415d4e455426bc22d9d1c1aa3691229bd35f0b62dc35af4f152dda0bf6285d77" Nov 03 16:53:43.904313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.904223 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq" Nov 03 16:53:43.953115 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:43.953049 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" podStartSLOduration=1.6562487620000002 podStartE2EDuration="1.953033974s" totalImagesPullingTime="296.785212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:43.948764171 +0000 UTC m=+2367.642035193" watchObservedRunningTime="2026-11-03 16:53:43.953033974 +0000 UTC m=+2367.646304995" Nov 03 16:53:44.340905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:44.340875 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:45.971814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:45.971777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:47.604167 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:47.604118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:49.098225 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.098191 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg"] Nov 03 16:53:49.098741 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.098726 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5c9c1086-d742-4cce-b02a-ed514d30cb72" containerName="revoke-keys" Nov 03 16:53:49.098783 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.098744 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9c1086-d742-4cce-b02a-ed514d30cb72" containerName="revoke-keys" Nov 03 16:53:49.098851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.098842 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9c1086-d742-4cce-b02a-ed514d30cb72" containerName="revoke-keys" Nov 03 16:53:49.102417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.102395 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.126866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.126833 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh"] Nov 03 16:53:49.130843 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.130772 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.137865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.137841 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg"] Nov 03 16:53:49.158727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.158694 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh"] Nov 03 16:53:49.166342 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166299 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166446 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166519 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166611 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166672 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfg5\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166802 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166778 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166843 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166827 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166878 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.166928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.166916 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.181923 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.181889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:49.268358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268322 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.268358 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268365 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.268677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268471 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.268677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268501 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.268677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268533 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.268677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268559 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.268677 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268657 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.268950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268739 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.268950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268764 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.268950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268801 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfg5\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.268950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268919 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.268969 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269022 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.269155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269050 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269079 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269095 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269094 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269319 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269523 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.269604 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269576 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.269666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269615 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjmfm\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.269800 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.269780 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.270413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.270376 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.271789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.271762 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.272708 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.272673 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.290548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.290519 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.290755 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.290734 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfg5\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.370896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.370807 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.370896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.370861 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.370896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.370891 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.370911 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.370947 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371033 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371121 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371198 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371170 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjmfm\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371207 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371285 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371479 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371378 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371594 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371626 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.371969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.371902 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.373549 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.373522 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.373806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.373785 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.392274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.392248 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.392385 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.392373 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjmfm\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.416836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.416714 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:49.446488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.446454 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:49.738691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.738644 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg"] Nov 03 16:53:49.740145 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:49.740114 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3db4b84_7faf_49de_b301_c3143c8c9b88.slice/crio-7803efc3b1fa2012139e8ba0f603146d70b35eca0bdd61389638af5d53c10d5e WatchSource:0}: Error finding container 7803efc3b1fa2012139e8ba0f603146d70b35eca0bdd61389638af5d53c10d5e: Status 404 returned error can't find the container with id 7803efc3b1fa2012139e8ba0f603146d70b35eca0bdd61389638af5d53c10d5e Nov 03 16:53:49.743192 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.743156 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:49.743297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.743243 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:49.743297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.743287 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:49.776118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.776086 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh"] Nov 03 16:53:49.776780 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:53:49.776739 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc2e931_cc73_4068_ab47_f7c8a7eef053.slice/crio-8137222ac94d844a623ed27f8696758e5a1105c31a0615a0022e0f67cbbf5895 WatchSource:0}: Error finding container 8137222ac94d844a623ed27f8696758e5a1105c31a0615a0022e0f67cbbf5895: Status 404 returned error can't find the container with id 8137222ac94d844a623ed27f8696758e5a1105c31a0615a0022e0f67cbbf5895 Nov 03 16:53:49.779843 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.779805 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:49.779956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.779893 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:49.779956 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.779935 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:49.933617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.932738 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:53:49.947832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.947504 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" event={"ID":"1dc2e931-cc73-4068-ab47-f7c8a7eef053","Type":"ContainerStarted","Data":"46225b336e954a4df9e7d0db994a85b7f33836599a5ed29c8469a179efc7a913"} Nov 03 16:53:49.947832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.947567 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" event={"ID":"1dc2e931-cc73-4068-ab47-f7c8a7eef053","Type":"ContainerStarted","Data":"8137222ac94d844a623ed27f8696758e5a1105c31a0615a0022e0f67cbbf5895"} Nov 03 16:53:49.952489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.952402 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" event={"ID":"a3db4b84-7faf-49de-b301-c3143c8c9b88","Type":"ContainerStarted","Data":"cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055"} Nov 03 16:53:49.952489 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:49.952486 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" event={"ID":"a3db4b84-7faf-49de-b301-c3143c8c9b88","Type":"ContainerStarted","Data":"7803efc3b1fa2012139e8ba0f603146d70b35eca0bdd61389638af5d53c10d5e"} Nov 03 16:53:50.047585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.046908 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podStartSLOduration=1.04688787 podStartE2EDuration="1.04688787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:50.04218675 +0000 UTC m=+2373.735457794" watchObservedRunningTime="2026-11-03 16:53:50.04688787 +0000 UTC m=+2373.740158893" Nov 03 16:53:50.119765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.119701 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podStartSLOduration=1.1196806480000001 podStartE2EDuration="1.119680648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:50.11422007 +0000 UTC m=+2373.807491094" watchObservedRunningTime="2026-11-03 16:53:50.119680648 +0000 UTC m=+2373.812951670" Nov 03 16:53:50.417337 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.417203 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:53:50.419134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.419090 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:50.419296 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.419152 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:50.446727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.446698 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:53:50.448155 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.448130 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:50.448277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:50.448182 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:51.019450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:51.013546 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:51.419579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:51.419540 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:51.420283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:51.420245 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:51.450592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:51.450551 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:51.450862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:51.450827 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:52.423452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:52.419569 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:52.423452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:52.419630 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:52.451458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:52.450837 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:52.451458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:52.450900 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:52.657451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:52.649861 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:53.417808 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:53.417772 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:53.418093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:53.418065 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:53.448242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:53.448200 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:53.448809 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:53.448780 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:53.990196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:53.990168 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6"] Nov 03 16:53:53.995753 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:53.995728 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.053987 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.049364 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6"] Nov 03 16:53:54.090085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.087493 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g"] Nov 03 16:53:54.092653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.092626 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.092860 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.092839 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g"] Nov 03 16:53:54.093300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093271 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093313 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjfv6\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093406 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093357 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093409 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093484 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093653 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093780 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093879 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093863 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.093939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.093926 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.194935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.194897 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.194935 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.194937 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.194999 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195031 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195052 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195076 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195116 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdxm\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195143 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195165 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195196 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195218 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195236 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195262 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195290 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195306 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjfv6\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195329 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.195550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195351 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.195550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.195389 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.196397 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.196376 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.196910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.196882 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.197021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.197003 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.197164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.197112 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.197261 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.197241 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.203482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.199099 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.203482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.199238 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.263499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.262461 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.263499 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.262758 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjfv6\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.296972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.296934 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.296972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.296974 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297025 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdxm\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297053 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297087 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297101 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297124 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297151 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297200 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297443 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297490 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.297863 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297815 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.298008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.297917 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.298204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.298185 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.322458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.307996 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.322458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.321276 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.364452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.344193 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdxm\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.365385 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.365316 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.369880 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.369812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:54.396949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.396920 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:54.459544 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.459512 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:54.460039 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.460013 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:54.460147 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.460061 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:54.480763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.479799 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:54.480763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.479885 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:54.982243 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.981586 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6"] Nov 03 16:53:54.984267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.984223 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:54.984380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.984362 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:54.984491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.984468 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:54.986017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.985990 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g"] Nov 03 16:53:54.987907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.987861 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:54.988021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.987997 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:54.988104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.988086 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:53:54.997769 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.997742 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" event={"ID":"23f605cc-337c-4145-b4f4-a18dd70f06bf","Type":"ContainerStarted","Data":"af95c639fb87520ed55311bda7d7a3aa3677ff36b910426f73dd5845876f7e2b"} Nov 03 16:53:54.998925 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:54.998901 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" event={"ID":"049c1081-44d2-440e-91bd-ec008d36c2bb","Type":"ContainerStarted","Data":"d0c2cceeeed10e878e8b5568a7ff3e57d0572c3df0b898f3f6274418b351a58f"} Nov 03 16:53:55.421459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:55.420570 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:55.425452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:55.424473 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:55.451460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:55.447825 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:55.451460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:55.447882 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:56.088209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.084353 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" event={"ID":"23f605cc-337c-4145-b4f4-a18dd70f06bf","Type":"ContainerStarted","Data":"ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a"} Nov 03 16:53:56.088209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.087377 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" event={"ID":"049c1081-44d2-440e-91bd-ec008d36c2bb","Type":"ContainerStarted","Data":"a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b"} Nov 03 16:53:56.162057 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.161906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:56.237452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.214927 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podStartSLOduration=3.214907015 podStartE2EDuration="3.214907015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:56.202332224 +0000 UTC m=+2379.895603276" watchObservedRunningTime="2026-11-03 16:53:56.214907015 +0000 UTC m=+2379.908178039" Nov 03 16:53:56.284470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.283956 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podStartSLOduration=3.283939988 podStartE2EDuration="3.283939988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:56.283737625 +0000 UTC m=+2379.977008646" watchObservedRunningTime="2026-11-03 16:53:56.283939988 +0000 UTC m=+2379.977211010" Nov 03 16:53:56.294316 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:53:56.294280 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:53:56.408515 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.407471 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:53:56.422450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.409780 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.422450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.409833 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:53:56.443452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.431227 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.443452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.431281 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:56.450450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.449838 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.450450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.449897 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:56.474450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.468889 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:53:56.474450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.472551 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.474450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:56.472608 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:53:57.397606 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.397517 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.398209 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.398177 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:53:57.448453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.444415 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.448453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.444504 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 16:53:57.448453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.448077 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.448453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.448121 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 16:53:57.496603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.482086 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.496603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.496517 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:53:57.542852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:57.542810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:58.246221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.243160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:58.407414 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.407330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:58.433862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.432992 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:53:58.433862 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.433050 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:53:58.464702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.461964 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:53:58.464702 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.462019 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:53:58.776846 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:58.769906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:53:59.166337 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.165575 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:59.411392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.410916 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:53:59.411392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.410985 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:53:59.468417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.462345 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:53:59.468417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.462401 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:53:59.468417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.467995 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:53:59.468417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.468045 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:59.496464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.490522 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:53:59.496464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.490584 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:59.500657 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:53:59.499800 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:00.421451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.416625 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:00.421451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.416687 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:00.468451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.467922 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:00.468451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.467985 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:00.478449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.476525 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:00.478449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.476592 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:00.623592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.623546 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:00.623910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.623867 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:00.982024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.981992 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:54:00.983401 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:00.983248 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:54:01.004538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.004491 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 16:54:01.298592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.298532 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.404455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.403986 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:01.404455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.404050 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:01.462877 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.461553 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.485724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.485683 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:01.485939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.485745 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:01.486159 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.486129 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.568319 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.568210 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.648749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.648709 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:01.649024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:01.648993 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.011393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.006318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:02.307452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.301551 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.332451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.320600 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:54:02.407392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.407357 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:02.408724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.408690 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:02.544660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.540892 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:02.544660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.540960 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:02.576704 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.575769 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 16:54:02.711117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.711026 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:02.724791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.712547 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:02.724791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.712606 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.748841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.740576 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:54:02.974090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:02.973988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:03.099573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.099520 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:54:03.201462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.191020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:03.325827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.325776 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:54:03.331510 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.331474 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg"] Nov 03 16:54:03.410447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.399880 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:03.410447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.399939 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:03.435641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.411001 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.436359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.422912 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:03.444546 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.444504 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:03.475481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.472302 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:03.475481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.472372 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:03.529451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.514230 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.529782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.529747 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.555011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.554975 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.587349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.587263 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.587637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.587614 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.587797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.587782 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.587934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.587918 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7mp\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.588089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.588076 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.613724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.613676 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.614100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.492599 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:03.614637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.554868 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p"] Nov 03 16:54:03.634621 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.587136 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:03.678836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.678808 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.737967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.737929 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.738281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.738259 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.738409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.738393 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.738567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.738550 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.738699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.738685 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.738825 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.738810 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.738945 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.738931 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.739100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.739084 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7mp\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.739265 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.739247 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.789797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.741046 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.789797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.741308 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.789797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.741553 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.789797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.742150 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.789797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.745134 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.789797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.765508 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:03.836774 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.814867 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg"] Nov 03 16:54:03.837483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.837394 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:03.837834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.837806 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:03.844102 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.841286 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wp8m\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.866750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.844301 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.867139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.867081 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.867600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.867567 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.867831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.867773 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.867986 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.867940 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.868142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.867970 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.868297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.868230 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.868597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.868501 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.947521 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.947410 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.957829 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.957794 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.970302 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.970260 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.970683 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.970661 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wp8m\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.970838 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.970823 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.970998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.970981 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.971173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.971159 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.971310 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.971296 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.971485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.971467 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.971626 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.971612 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.971753 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.971740 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.972313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.972289 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.973978 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.973954 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.974933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.974912 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.989682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.985945 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:03.989682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.986600 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:03.989682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:03.988519 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:04.081373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.079635 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7mp\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:04.081373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.079706 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p"] Nov 03 16:54:04.107551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.103699 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:04.163628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.163579 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:54:04.170474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.146826 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:04.242230 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.242190 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:04.437289 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.437200 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:04.527933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.441812 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:04.580901 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.473862 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:54:04.627689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.617121 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:04.627689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.619895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:04.684998 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.502940 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:54:04.687563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.508182 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wp8m\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:04.687563 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.687254 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:04.688196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.688174 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:54:04.738590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.738538 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 16:54:04.783968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.783919 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:04.815286 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.815239 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:04.827596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.821505 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:04.827596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.822212 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:04.827596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.822256 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:04.853187 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.853148 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:04.941086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.940994 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:04.957724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.950779 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:54:04.957724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.953226 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:04.957724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.953281 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:04.962492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:04.962448 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:05.179461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.176925 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:54:05.466450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.463846 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:05.466450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.463930 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:05.525045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.525008 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:05.525240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.525084 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:05.648467 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.641201 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:05.648467 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.641317 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:54:05.648467 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.641373 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 16:54:05.690930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.690872 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:54:05.906270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:05.861536 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:06.080036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.079630 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:06.080036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.079831 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:54:06.119491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.118745 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:06.119491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.118853 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:06.235452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.206698 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:06.618077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.611745 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.618077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.611812 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:06.760882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.760714 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:06.900583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.874283 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:06.900583 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.874983 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:06.919036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.860578 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:06.919036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.906192 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:06.919036 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.906310 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:06.947548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.920505 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:54:06.947548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.937643 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.947548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.937701 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:06.947548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:06.937785 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:54:07.087173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.087121 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 16:54:07.087528 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.087509 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:54:07.087681 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.087669 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:07.087793 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.087779 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg"] Nov 03 16:54:07.137878 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.127906 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:54:07.393720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.392619 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:07.393720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.392702 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:07.393720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.392747 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:07.400795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.400618 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.400795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.400668 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:07.467572 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.467540 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.467765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.467599 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:07.559488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.506261 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p"] Nov 03 16:54:07.578577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.578535 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:54:07.597840 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.597818 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:54:07.797733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.797649 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" event={"ID":"ca874712-8415-442c-892e-5bcdd0f0bffc","Type":"ContainerStarted","Data":"206eddb5e239b98f8f0475611844aac3887459feec77f396995b73c43b5f7535"} Nov 03 16:54:07.816608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.816401 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" event={"ID":"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a","Type":"ContainerStarted","Data":"4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67"} Nov 03 16:54:07.816608 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.816489 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" event={"ID":"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a","Type":"ContainerStarted","Data":"299fdafd9d770e68f4a44f39a6e53568f4b0adcd6c5d5cd53b365365eae93a1f"} Nov 03 16:54:07.877886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:07.875812 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podStartSLOduration=4.875789495 podStartE2EDuration="4.875789495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:07.866396229 +0000 UTC m=+2391.559667252" watchObservedRunningTime="2026-11-03 16:54:07.875789495 +0000 UTC m=+2391.569060517" Nov 03 16:54:08.338494 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.337373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:08.403584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.398547 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.403584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.398604 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:08.481951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.481884 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.482156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.481963 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 16:54:08.699759 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.699668 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:08.706791 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.706754 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.707102 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:08.707072 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:09.043542 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.043504 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c"] Nov 03 16:54:09.129450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.102105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:09.129450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.104995 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.129450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.116863 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" event={"ID":"ca874712-8415-442c-892e-5bcdd0f0bffc","Type":"ContainerStarted","Data":"5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353"} Nov 03 16:54:09.133679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.133645 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c"] Nov 03 16:54:09.136242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.136109 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3b6f35-openshift-default-dockercfg-jr4pt\"" Nov 03 16:54:09.252446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.245014 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podStartSLOduration=6.244991764 podStartE2EDuration="6.244991764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:09.214266175 +0000 UTC m=+2392.907537198" watchObservedRunningTime="2026-11-03 16:54:09.244991764 +0000 UTC m=+2392.938262781" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299050 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299160 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299233 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299277 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299322 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299353 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvqf\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299405 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299528 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.302512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.299577 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.306139 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.306110 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:09.399151 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.398571 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401179 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401249 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401396 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401464 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401541 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401580 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401635 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401759 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvqf\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401831 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401871 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 16:54:09.404614 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.401911 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.405512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.404503 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.405512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.404783 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.405799 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.405774 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.434518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.406584 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.434801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.416207 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.434933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.423858 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.478446 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.467408 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvqf\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.483497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.481853 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:54:09.501458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.488008 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.554207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.554117 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:09.578027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.577977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:09.700629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.700277 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.700629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.700344 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:09.702907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.702610 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:09.702907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.702802 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:09.703224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.703204 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.703836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.703776 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:09.810504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:09.809636 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:10.095558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.091534 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:54:10.095558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.092700 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:10.486511 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.465820 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 16:54:10.512450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.502539 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:10.512450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.502612 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:10.516468 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.514327 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:10.699480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.697270 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.699480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.697334 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:10.699480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.697952 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.699480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.697986 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:10.714501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.713627 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:10.714501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.714051 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:54:10.812451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.806977 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c"] Nov 03 16:54:10.927991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.927928 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:10.928345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.928316 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:10.938908 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:10.938869 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:11.052991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.052961 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:54:11.077065 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.077023 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 16:54:11.104900 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.102058 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:11.104900 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.102473 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 03 16:54:11.105979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.105323 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:54:11.105979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.105395 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:11.415883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.412530 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:11.522892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.426543 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:11.522892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.426611 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:11.522892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.444202 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:11.557891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.547414 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:54:11.696922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.696804 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.697097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.696903 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:11.699064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.697788 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.699064 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.697838 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:11.710083 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.710057 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 16:54:11.713220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.713186 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" event={"ID":"93bc673a-89ea-43c1-85bd-57952e0c2848","Type":"ContainerStarted","Data":"24e0776a4034920640ec5fdffc2067824d0e8aafa322fb65fa109a0f4aea901b"} Nov 03 16:54:11.714257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.714226 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:11.749968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.749060 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:54:11.853498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.853468 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:11.903508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:11.880780 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:54:12.260492 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.252316 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:12.693369 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.693335 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:12.693637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.693609 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:12.699340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.699312 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:12.699590 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.699566 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:12.771629 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.767770 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" event={"ID":"93bc673a-89ea-43c1-85bd-57952e0c2848","Type":"ContainerStarted","Data":"2253d66a488590b83b756961622ac959a6c3317b2e69fb73a64f242d2b69b23e"} Nov 03 16:54:12.886538 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.881969 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podStartSLOduration=4.881946975 podStartE2EDuration="4.881946975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:12.880963595 +0000 UTC m=+2396.574234653" watchObservedRunningTime="2026-11-03 16:54:12.881946975 +0000 UTC m=+2396.575217998" Nov 03 16:54:12.887602 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.887522 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:12.927070 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:12.926770 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:13.241512 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.240524 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:13.253462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.252629 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:13.331578 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.330528 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:54:13.464195 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.463535 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:13.713500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.688821 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:13.713500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.688882 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:13.713500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.707450 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:13.713500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.707534 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:13.723968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.723925 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:13.905699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.905611 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:13.907796 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.907721 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:13.935827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:13.935515 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:14.363451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.330818 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 16:54:14.372580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.367894 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:54:14.376465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.339129 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:14.376465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.374280 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:54:14.396097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.396070 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:54:14.493501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.472547 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:14.493501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.473324 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:14.690482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.690374 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:54:14.693502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.693473 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:14.698810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.698768 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:14.699093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.694123 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:14.767520 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.699315 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:14.869040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.868988 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:14.869338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.796549 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:14.878455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:14.869487 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:15.018504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.008106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:15.328451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.316363 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:15.488086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.488037 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:15.689724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.689645 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:15.689724 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.689698 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:15.697331 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.697302 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:15.697506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:15.697360 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:16.660160 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.660078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:16.689245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.689209 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:16.689458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.689270 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:16.698447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.697680 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 16:54:16.698447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.697739 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 16:54:16.816930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.816884 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:16.915308 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.915229 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:54:16.920281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:16.920259 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:54:17.086645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:17.086610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:17.366961 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:17.366880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:17.689281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:17.689244 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 16:54:17.689841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:17.689304 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 16:54:17.829475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:17.827036 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:17.978465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:17.978335 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:18.195443 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.195382 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:18.196648 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.196624 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:18.197552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.197528 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 16:54:18.758372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.758340 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:18.758930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.758828 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:18.759808 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.759784 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 16:54:18.868680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:18.868650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:19.039832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:19.037297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:19.193700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:19.193665 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:20.410585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:20.407650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:21.032686 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.032648 4539 generic.go:356] "Generic (PLEG): container finished" podID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerID="08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45" exitCode=0 Nov 03 16:54:21.037233 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.033008 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" event={"ID":"c50b512a-6cf5-40be-9d89-893bd82072ad","Type":"ContainerDied","Data":"08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45"} Nov 03 16:54:21.039454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.039407 4539 scope.go:122] "RemoveContainer" containerID="08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45" Nov 03 16:54:21.042990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.041587 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:21.042990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.041679 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:21.042990 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.041731 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:21.166457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.156857 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:21.654158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.654069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:21.866507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:21.865394 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:22.067651 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.066578 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" event={"ID":"c50b512a-6cf5-40be-9d89-893bd82072ad","Type":"ContainerStarted","Data":"93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485"} Nov 03 16:54:22.094970 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.094928 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:54:22.095363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.095337 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:22.095601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.095575 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:22.147493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.147350 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:22.399526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.397524 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.612356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.610856 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.700624 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.700499 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:22.782448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.775275 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.992564 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:22.938863 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:54:23.111447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.111229 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:23.111447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.111307 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:23.290985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.287996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:23.461750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.459338 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.461750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.459765 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:54:23.471916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.471838 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:23.615882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.612531 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.615882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.614173 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:54:23.879950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.879841 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.884449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.881243 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:24.005457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:23.998816 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:24.062448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.046930 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:54:24.288004 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.287864 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:24.294675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.288043 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:24.483458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.482141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:24.483458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.482275 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:54:24.629597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.626445 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:24.662456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.631220 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 16:54:24.662456 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.648523 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:24.768102 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.768044 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:24.857461 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:24.855341 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 16:54:25.017933 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.017840 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 16:54:25.100245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.100210 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:25.100625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.100563 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:25.142799 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.142757 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 16:54:25.275264 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.275238 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 16:54:25.489274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.489227 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 16:54:25.786688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.786659 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 16:54:25.794340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.794299 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:25.795287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.795124 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:54:25.842545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.842048 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 16:54:25.852939 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:25.852906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:26.015449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:26.011761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:26.093910 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:26.093819 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:26.094556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:26.094521 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:26.310449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:26.307400 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:26.348264 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:54:26.348049 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:54:26.574584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:26.574548 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:26.995551 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:26.995522 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:27.092024 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:27.091986 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:27.092219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:27.092044 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:27.211472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:27.207235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:27.836866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:27.836820 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:28.024451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.021248 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:28.091818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.091731 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:28.091818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.091786 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:28.190482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.190415 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:28.345816 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.345699 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:28.528449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.528376 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:28.662691 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.662605 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:54:28.851789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:28.851723 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:29.048580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:29.048541 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:29.092518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:29.092477 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:29.092676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:29.092541 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:29.376799 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:29.376657 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:30.092184 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:30.092143 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:30.092388 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:30.092205 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:30.477277 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:30.477167 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:30.892095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:30.892065 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:31.064463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:31.064397 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:31.093398 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:31.093364 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 16:54:31.093789 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:31.093739 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 16:54:32.059478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.059442 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92"] Nov 03 16:54:32.065118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.065094 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.097740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.097707 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92"] Nov 03 16:54:32.107767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.107738 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:54:32.109661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.109638 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:54:32.132264 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.132193 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-3b6f35-serving-cert\"" Nov 03 16:54:32.167118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.167073 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.167340 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.167187 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.167507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.167488 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.167610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.167514 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96jq\" (UniqueName: \"kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.175403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.175367 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:32.269831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.269721 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.270120 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.270093 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96jq\" (UniqueName: \"kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.270336 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.270317 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.270566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.270386 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.271643 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.271618 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.273767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.273738 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.273902 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.273879 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.337450 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.337318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:32.357051 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.356977 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96jq\" (UniqueName: \"kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq\") pod \"maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.381838 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.381789 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:32.708903 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:54:32.708869 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400a9d3e_8c3a_4669_94a2_4642adbe2525.slice/crio-a00d08b8ea271d919fea7a08258baffb0afcdc1f1bd47ec16651ab83ebeb1bf1 WatchSource:0}: Error finding container a00d08b8ea271d919fea7a08258baffb0afcdc1f1bd47ec16651ab83ebeb1bf1: Status 404 returned error can't find the container with id a00d08b8ea271d919fea7a08258baffb0afcdc1f1bd47ec16651ab83ebeb1bf1 Nov 03 16:54:32.709050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:32.709002 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92"] Nov 03 16:54:33.617735 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:33.617701 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" event={"ID":"400a9d3e-8c3a-4669-94a2-4642adbe2525","Type":"ContainerStarted","Data":"23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e"} Nov 03 16:54:33.618242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:33.617753 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" event={"ID":"400a9d3e-8c3a-4669-94a2-4642adbe2525","Type":"ContainerStarted","Data":"a00d08b8ea271d919fea7a08258baffb0afcdc1f1bd47ec16651ab83ebeb1bf1"} Nov 03 16:54:33.618242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:33.617784 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:33.663178 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:33.663118 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" podStartSLOduration=1.382072536 podStartE2EDuration="1.66310018s" totalImagesPullingTime="281.027644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:33.659083329 +0000 UTC m=+2417.352354355" watchObservedRunningTime="2026-11-03 16:54:33.66310018 +0000 UTC m=+2417.356371206" Nov 03 16:54:33.992972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:33.992892 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:34.712288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:34.712240 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:34.865032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:34.864995 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:35.014747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:35.014639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:36.089979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:36.089940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:37.335356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.335314 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q"] Nov 03 16:54:37.340537 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.340514 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.355220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.355198 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 16:54:37.355220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.355215 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 16:54:37.355366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.355329 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 16:54:37.364693 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.364642 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xzr7j\"" Nov 03 16:54:37.370259 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.370235 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q"] Nov 03 16:54:37.460794 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.460749 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.461015 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.460893 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.461015 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.460951 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.461111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.461058 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwffc\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.461165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.461109 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.461233 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.461211 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.461292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.461261 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.562991 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwffc\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.563064 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.563102 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.563131 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.563354 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563527 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.563487 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.563639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.563545 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.564197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.564168 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.565784 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.565757 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.565882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.565781 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.566283 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.566259 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.566372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.566317 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.584842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.584815 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwffc\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.584964 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.584817 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxw8q\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.654349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.654266 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:37.716903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.709399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:37.999098 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:54:37.999052 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77dbf16b_0eb3_4043_a796_b3380d4ef3ce.slice/crio-92ac7ea27e3aea41ed591979762498f8e40baf95f7c2c6c152435e4feccf0ec5 WatchSource:0}: Error finding container 92ac7ea27e3aea41ed591979762498f8e40baf95f7c2c6c152435e4feccf0ec5: Status 404 returned error can't find the container with id 92ac7ea27e3aea41ed591979762498f8e40baf95f7c2c6c152435e4feccf0ec5 Nov 03 16:54:37.999926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:37.999887 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q"] Nov 03 16:54:38.001949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:38.001919 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:38.002021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:38.001980 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:38.668625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:38.668514 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" event={"ID":"77dbf16b-0eb3-4043-a796-b3380d4ef3ce","Type":"ContainerStarted","Data":"c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e"} Nov 03 16:54:38.668625 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:38.668583 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" event={"ID":"77dbf16b-0eb3-4043-a796-b3380d4ef3ce","Type":"ContainerStarted","Data":"92ac7ea27e3aea41ed591979762498f8e40baf95f7c2c6c152435e4feccf0ec5"} Nov 03 16:54:38.669731 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:38.669620 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:38.739175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:38.739120 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" podStartSLOduration=1.739105277 podStartE2EDuration="1.739105277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:38.72937577 +0000 UTC m=+2422.422646794" watchObservedRunningTime="2026-11-03 16:54:38.739105277 +0000 UTC m=+2422.432376296" Nov 03 16:54:39.497853 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:39.497817 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:39.636290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:39.636251 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 16:54:39.678197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:39.678170 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 16:54:41.208487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:41.208441 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:41.349095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:41.349040 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:43.030469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:43.019021 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:43.195848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:43.195810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:43.348506 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:43.348359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:43.653928 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:43.653833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:44.204095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:44.204060 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:44.641849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:44.641818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:45.662516 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:45.662480 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:47.016458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:47.008574 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:47.016458 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:47.008666 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:47.564382 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:47.564350 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:47.738449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:47.736928 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:48.129449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:48.129382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:48.296100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:48.296064 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:49.565778 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:49.565745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:49.778052 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:49.777939 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:50.764592 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:50.764546 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:50.934775 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:50.934744 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:51.696666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:51.696633 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:53.262556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:53.262524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:54.500348 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:54.500318 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k"] Nov 03 16:54:54.501052 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:54.501022 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" podUID="601584a0-5ff2-4aa6-81bd-95c8115cac17" containerName="maas-api" containerID="cri-o://d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787" gracePeriod=30 Nov 03 16:54:54.603812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:54.603741 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:55.186918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.186791 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:54:55.190703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.190675 4539 generic.go:356] "Generic (PLEG): container finished" podID="601584a0-5ff2-4aa6-81bd-95c8115cac17" containerID="d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787" exitCode=0 Nov 03 16:54:55.190852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.190722 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" Nov 03 16:54:55.190852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.190760 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" event={"ID":"601584a0-5ff2-4aa6-81bd-95c8115cac17","Type":"ContainerDied","Data":"d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787"} Nov 03 16:54:55.190852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.190813 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" event={"ID":"601584a0-5ff2-4aa6-81bd-95c8115cac17","Type":"ContainerDied","Data":"203dca0ae0d42b0ab34025a5b491098756b5c559a0d6891386d10ea33e046d11"} Nov 03 16:54:55.190852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.190837 4539 scope.go:122] "RemoveContainer" containerID="d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787" Nov 03 16:54:55.191323 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.191308 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca\") pod \"601584a0-5ff2-4aa6-81bd-95c8115cac17\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " Nov 03 16:54:55.191397 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.191341 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls\") pod \"601584a0-5ff2-4aa6-81bd-95c8115cac17\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " Nov 03 16:54:55.191497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.191481 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs\") pod \"601584a0-5ff2-4aa6-81bd-95c8115cac17\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " Nov 03 16:54:55.191556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.191534 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb\" (UniqueName: \"kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb\") pod \"601584a0-5ff2-4aa6-81bd-95c8115cac17\" (UID: \"601584a0-5ff2-4aa6-81bd-95c8115cac17\") " Nov 03 16:54:55.191737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.191716 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca" pod "601584a0-5ff2-4aa6-81bd-95c8115cac17" (UID: "601584a0-5ff2-4aa6-81bd-95c8115cac17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:55.192453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.192382 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/601584a0-5ff2-4aa6-81bd-95c8115cac17-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:54:55.194117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.194081 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls" pod "601584a0-5ff2-4aa6-81bd-95c8115cac17" (UID: "601584a0-5ff2-4aa6-81bd-95c8115cac17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:55.194272 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.194254 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs" pod "601584a0-5ff2-4aa6-81bd-95c8115cac17" (UID: "601584a0-5ff2-4aa6-81bd-95c8115cac17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:55.196454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.196402 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb" pod "601584a0-5ff2-4aa6-81bd-95c8115cac17" (UID: "601584a0-5ff2-4aa6-81bd-95c8115cac17"). InnerVolumeSpecName "kube-api-access-9pxqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:55.261854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.261630 4539 scope.go:122] "RemoveContainer" containerID="d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787" Nov 03 16:54:55.262488 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:54:55.262459 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787\": container with ID starting with d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787 not found: ID does not exist" containerID="d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787" Nov 03 16:54:55.262597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.262502 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787"} err="failed to get container status \"d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787\": rpc error: code = NotFound desc = could not find container \"d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787\": container with ID starting with d18f9dadd0a4772e7da38009f808fa14159ca4819a62e9824ef86d6a3af73787 not found: ID does not exist" Nov 03 16:54:55.303685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.303647 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:54:55.303685 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.303680 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/601584a0-5ff2-4aa6-81bd-95c8115cac17-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:54:55.303937 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.303694 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pxqb\" (UniqueName: \"kubernetes.io/projected/601584a0-5ff2-4aa6-81bd-95c8115cac17-kube-api-access-9pxqb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:54:55.535449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.535407 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k"] Nov 03 16:54:55.546117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.546088 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k"] Nov 03 16:54:55.592041 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:55.592009 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:56.157285 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:54:56.157239 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:54:56.200165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.200134 4539 generic.go:356] "Generic (PLEG): container finished" podID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerID="2253d66a488590b83b756961622ac959a6c3317b2e69fb73a64f242d2b69b23e" exitCode=0 Nov 03 16:54:56.200366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.200207 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" event={"ID":"93bc673a-89ea-43c1-85bd-57952e0c2848","Type":"ContainerDied","Data":"2253d66a488590b83b756961622ac959a6c3317b2e69fb73a64f242d2b69b23e"} Nov 03 16:54:56.200906 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.200881 4539 scope.go:122] "RemoveContainer" containerID="2253d66a488590b83b756961622ac959a6c3317b2e69fb73a64f242d2b69b23e" Nov 03 16:54:56.202727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.202692 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:56.202851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.202788 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:56.202851 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.202834 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 16:54:56.349730 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.349698 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:56.471772 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.471673 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601584a0-5ff2-4aa6-81bd-95c8115cac17" path="/var/lib/kubelet/pods/601584a0-5ff2-4aa6-81bd-95c8115cac17/volumes" Nov 03 16:54:56.910449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:56.910143 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:57.110394 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.108036 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:57.248500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.247582 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" event={"ID":"93bc673a-89ea-43c1-85bd-57952e0c2848","Type":"ContainerStarted","Data":"96e6fd4363ceec7ff9187cacb19a72daead427afb21ac9da56964a6a01968db7"} Nov 03 16:54:57.264449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.260809 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:57.470447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.469462 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:57.748819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.748779 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:57.814854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.814823 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:57.818697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.818633 4539 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:54:57.819016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.818986 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 16:54:57.919556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:57.919510 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a11d94-849579fc-zbv8k" podUID="601584a0-5ff2-4aa6-81bd-95c8115cac17" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.125:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:58.089689 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.089640 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:58.356133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.356009 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl"] Nov 03 16:54:58.356642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.356609 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" containerID="cri-o://93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485" gracePeriod=30 Nov 03 16:54:58.431451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.430999 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:58.815579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.812836 4539 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:54:58.815579 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.812912 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 16:54:58.916454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:58.913468 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:54:59.716475 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:59.715180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:54:59.818886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:59.818846 4539 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:54:59.819186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:54:59.819157 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 16:55:00.344751 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:00.344721 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:00.517380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:00.517300 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:00.816460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:00.815909 4539 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:55:00.816460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:00.815993 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 16:55:01.481417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:01.481382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:01.668293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:01.668260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:01.817449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:01.811914 4539 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 16:55:01.817449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:01.811988 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 16:55:02.135671 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:02.135587 4539 patch_prober.go:29] interesting pod/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:55:02.140452 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:02.135887 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:55:02.146847 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:02.146418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:02.335295 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:02.335260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:02.865733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:02.865705 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:55:02.867069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:02.867049 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 16:55:03.376495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.376451 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:03.665166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.665145 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:55:03.815859 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.815823 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.815903 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816038 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.815944 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816038 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816072 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816181 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816094 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816221 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816301 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:55:03.816347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816314 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:55:03.816347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816342 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816589 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816468 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert\") pod \"c50b512a-6cf5-40be-9d89-893bd82072ad\" (UID: \"c50b512a-6cf5-40be-9d89-893bd82072ad\") " Nov 03 16:55:03.816650 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816592 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:55:03.816738 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816715 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:55:03.817021 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.816999 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:55:03.817276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.817162 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.817276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.817184 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.817276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.817198 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.817276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.817213 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c50b512a-6cf5-40be-9d89-893bd82072ad-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.817276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.817244 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.819053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.819015 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:55:03.819053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.819039 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "kube-api-access-lx547". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:55:03.819213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.819121 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:55:03.819213 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.819172 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token" pod "c50b512a-6cf5-40be-9d89-893bd82072ad" (UID: "c50b512a-6cf5-40be-9d89-893bd82072ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:55:03.918719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.918642 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.918719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.918675 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.918719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.918689 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:03.918719 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:03.918704 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx547\" (UniqueName: \"kubernetes.io/projected/c50b512a-6cf5-40be-9d89-893bd82072ad-kube-api-access-lx547\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:55:04.353517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.353483 4539 generic.go:356] "Generic (PLEG): container finished" podID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerID="93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485" exitCode=0 Nov 03 16:55:04.353700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.353537 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" Nov 03 16:55:04.353700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.353571 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" event={"ID":"c50b512a-6cf5-40be-9d89-893bd82072ad","Type":"ContainerDied","Data":"93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485"} Nov 03 16:55:04.353700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.353642 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl" event={"ID":"c50b512a-6cf5-40be-9d89-893bd82072ad","Type":"ContainerDied","Data":"eb050f108dae63b2d159448e5adbe508165ae422a9f7a7564b282cbb1c22449d"} Nov 03 16:55:04.353700 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.353664 4539 scope.go:122] "RemoveContainer" containerID="93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485" Nov 03 16:55:04.364366 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.364349 4539 scope.go:122] "RemoveContainer" containerID="08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45" Nov 03 16:55:04.373645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.373626 4539 scope.go:122] "RemoveContainer" containerID="93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485" Nov 03 16:55:04.373907 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:55:04.373888 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485\": container with ID starting with 93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485 not found: ID does not exist" containerID="93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485" Nov 03 16:55:04.373979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.373922 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485"} err="failed to get container status \"93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485\": rpc error: code = NotFound desc = could not find container \"93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485\": container with ID starting with 93a7a7f9d1968826aa64548c0c59d93fe914bc781d904f6a462379e34c8a4485 not found: ID does not exist" Nov 03 16:55:04.373979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.373950 4539 scope.go:122] "RemoveContainer" containerID="08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45" Nov 03 16:55:04.374206 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:55:04.374189 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45\": container with ID starting with 08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45 not found: ID does not exist" containerID="08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45" Nov 03 16:55:04.374251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.374213 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45"} err="failed to get container status \"08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45\": rpc error: code = NotFound desc = could not find container \"08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45\": container with ID starting with 08accd80957a6c8f3ff2d1a6f7c8bf2e4810f1dde602a776acc8e3ca75eacc45 not found: ID does not exist" Nov 03 16:55:04.443552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.443504 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl"] Nov 03 16:55:04.459191 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.457716 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a11d94-openshift-default-5d4567ccbf-kxcgl"] Nov 03 16:55:04.473554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:04.472939 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" path="/var/lib/kubelet/pods/c50b512a-6cf5-40be-9d89-893bd82072ad/volumes" Nov 03 16:55:05.051305 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:05.051263 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:06.834989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.834946 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t"] Nov 03 16:55:06.835548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835533 4539 cpu_manager.go:414] "containerMap: removing container" podUID="601584a0-5ff2-4aa6-81bd-95c8115cac17" containerName="maas-api" Nov 03 16:55:06.835623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835549 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601584a0-5ff2-4aa6-81bd-95c8115cac17" containerName="maas-api" Nov 03 16:55:06.835623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835565 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" Nov 03 16:55:06.835623 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835570 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" Nov 03 16:55:06.835716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835679 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="601584a0-5ff2-4aa6-81bd-95c8115cac17" containerName="maas-api" Nov 03 16:55:06.835716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835690 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" Nov 03 16:55:06.835716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.835698 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" Nov 03 16:55:06.840947 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.840930 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:06.856477 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.856457 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2b49df-serving-cert\"" Nov 03 16:55:06.877680 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.877649 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t"] Nov 03 16:55:06.949767 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.949730 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:06.972814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.972782 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:06.972989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.972829 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhrm\" (UniqueName: \"kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:06.972989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.972945 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:06.973144 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:06.973125 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.074930 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.074887 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.075112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.075046 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.075211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.075196 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhrm\" (UniqueName: \"kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.075299 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.075285 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.076577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.076547 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.084260 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.084230 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.117126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.111609 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhrm\" (UniqueName: \"kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.117126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.111715 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.117126 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.112233 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:07.160069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.160031 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:07.267380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.267353 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:07.468357 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:07.468327 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t"] Nov 03 16:55:07.468500 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:55:07.468362 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda139fc2b_2df8_44dd_9356_2cefc49d160d.slice/crio-201b13625b94b03e2d60fbdabad81b1d409001c8aec3d8a2d3904c8f9dbd3bc4 WatchSource:0}: Error finding container 201b13625b94b03e2d60fbdabad81b1d409001c8aec3d8a2d3904c8f9dbd3bc4: Status 404 returned error can't find the container with id 201b13625b94b03e2d60fbdabad81b1d409001c8aec3d8a2d3904c8f9dbd3bc4 Nov 03 16:55:08.045465 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.041660 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:08.384548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.384518 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" event={"ID":"a139fc2b-2df8-44dd-9356-2cefc49d160d","Type":"ContainerStarted","Data":"68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a"} Nov 03 16:55:08.384716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.384593 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" event={"ID":"a139fc2b-2df8-44dd-9356-2cefc49d160d","Type":"ContainerStarted","Data":"201b13625b94b03e2d60fbdabad81b1d409001c8aec3d8a2d3904c8f9dbd3bc4"} Nov 03 16:55:08.384716 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.384630 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:08.458723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.458649 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" podStartSLOduration=2.120311681 podStartE2EDuration="2.45862902s" totalImagesPullingTime="338.317339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:08.457638966 +0000 UTC m=+2452.150909989" watchObservedRunningTime="2026-11-03 16:55:08.45862902 +0000 UTC m=+2452.151900042" Nov 03 16:55:08.522664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.522623 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:08.701040 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.700941 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:08.860581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:08.860552 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:10.841055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:10.840902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:11.983013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:11.982972 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:12.352338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:12.352303 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:12.530819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:12.530743 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:12.730962 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:12.730919 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:12.964922 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:12.964825 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:13.110885 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:13.110796 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:13.788918 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:13.788880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:14.215742 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:14.215630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:14.369868 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:14.369807 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:14.398536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:14.398508 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 16:55:14.551750 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:14.551712 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:14.706676 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:14.706633 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:14.972955 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:14.972871 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:15.206281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:15.206238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:17.155487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:17.155449 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:17.400009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:17.399976 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:18.465484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:18.465453 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:18.903842 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:18.903811 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:19.046162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:19.046101 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:19.263259 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:19.263135 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:19.420085 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:19.420055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:19.591447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:19.589616 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:20.716174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:20.716134 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:21.160197 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.160153 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:21.279109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.279076 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts"] Nov 03 16:55:21.280000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.279977 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" Nov 03 16:55:21.280000 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.280001 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50b512a-6cf5-40be-9d89-893bd82072ad" containerName="istio-proxy" Nov 03 16:55:21.286239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.286217 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.306526 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.306412 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts"] Nov 03 16:55:21.323721 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.323692 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-c6f303-serving-cert\"" Nov 03 16:55:21.328050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.327994 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:21.358764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.358718 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.359016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.358858 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.359016 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.358960 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.359218 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.359018 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9wz\" (UniqueName: \"kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.460222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.460127 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.460413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.460286 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.460413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.460357 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9wz\" (UniqueName: \"kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.460647 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.460625 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.461480 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.461418 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.463988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.463961 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.464163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.463995 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.473957 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.473926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:21.484662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.484601 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9wz\" (UniqueName: \"kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz\") pod \"maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.603047 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.602711 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:21.643412 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:21.643238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:22.069912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:22.069877 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts"] Nov 03 16:55:22.071494 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:55:22.071467 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708707d8_acb2_4982_9c2c_52e48776d590.slice/crio-b51abe98039f85815c978240fce05572d7c9df5aba6330c51ec099bde76ffd41 WatchSource:0}: Error finding container b51abe98039f85815c978240fce05572d7c9df5aba6330c51ec099bde76ffd41: Status 404 returned error can't find the container with id b51abe98039f85815c978240fce05572d7c9df5aba6330c51ec099bde76ffd41 Nov 03 16:55:22.771983 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:22.771943 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" event={"ID":"708707d8-acb2-4982-9c2c-52e48776d590","Type":"ContainerStarted","Data":"238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea"} Nov 03 16:55:22.772171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:22.772011 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" event={"ID":"708707d8-acb2-4982-9c2c-52e48776d590","Type":"ContainerStarted","Data":"b51abe98039f85815c978240fce05572d7c9df5aba6330c51ec099bde76ffd41"} Nov 03 16:55:22.772171 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:22.772038 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:22.823584 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:22.823477 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" podStartSLOduration=1.369749013 podStartE2EDuration="1.823461178s" totalImagesPullingTime="453.712165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:22.810594903 +0000 UTC m=+2466.503865925" watchObservedRunningTime="2026-11-03 16:55:22.823461178 +0000 UTC m=+2466.516732200" Nov 03 16:55:23.264948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:23.264873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:24.862854 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:24.862824 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:26.157973 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:55:26.157931 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:55:26.481463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:26.481344 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:28.292125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:28.292079 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:28.787015 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:28.786983 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 16:55:30.055947 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:30.055908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:31.785063 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:31.785030 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:33.475979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:33.475942 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:34.865387 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.864916 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh"] Nov 03 16:55:34.870803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.870779 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:34.884806 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.884778 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-001802-serving-cert\"" Nov 03 16:55:34.906474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.906442 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh"] Nov 03 16:55:34.984698 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.984661 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:34.984916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.984713 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:34.984916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.984843 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8j45\" (UniqueName: \"kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:34.984916 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.984904 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:34.996500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:34.996459 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:35.086304 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.086267 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.086555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.086318 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.086555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.086464 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8j45\" (UniqueName: \"kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.086555 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.086531 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.087485 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.087454 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.089326 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.089275 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.089497 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.089477 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.108208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.108171 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8j45\" (UniqueName: \"kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45\") pod \"maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.165509 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.153985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:35.196573 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.194695 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:35.539205 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.539166 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh"] Nov 03 16:55:35.540791 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:55:35.540748 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf58250_f040_4089_9364_6600e22887ac.slice/crio-b815b9cf103fc9092408b632dd4a305e83c241f2aefa44bbfc0f416e9a07f00a WatchSource:0}: Error finding container b815b9cf103fc9092408b632dd4a305e83c241f2aefa44bbfc0f416e9a07f00a: Status 404 returned error can't find the container with id b815b9cf103fc9092408b632dd4a305e83c241f2aefa44bbfc0f416e9a07f00a Nov 03 16:55:35.928164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:35.928133 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" event={"ID":"dbf58250-f040-4089-9364-6600e22887ac","Type":"ContainerStarted","Data":"b815b9cf103fc9092408b632dd4a305e83c241f2aefa44bbfc0f416e9a07f00a"} Nov 03 16:55:36.109453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.091493 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84"] Nov 03 16:55:36.138503 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.135106 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.151290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.151243 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84"] Nov 03 16:55:36.152121 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.152096 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd067\"/\"kube-root-ca.crt\"" Nov 03 16:55:36.152491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.152472 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd067\"/\"default-dockercfg-bxdpz\"" Nov 03 16:55:36.152697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.152684 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd067\"/\"openshift-service-ca.crt\"" Nov 03 16:55:36.189715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.189618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:36.220723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.220691 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd067\"/\"auth-test-model-3dd067-kserve-self-signed-certs\"" Nov 03 16:55:36.236211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.236176 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wssj\" (UniqueName: \"kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.236418 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.236317 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.236554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.236418 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.236554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.236497 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.236554 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.236544 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.236710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.236606 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.337912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.337869 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.338106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.337977 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.338106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.338016 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.338106 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.338061 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.338294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.338118 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.338294 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.338224 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wssj\" (UniqueName: \"kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.338836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.338806 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.339270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.339240 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.339759 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.339697 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.341617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.341592 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.341734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.341713 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.370309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.360980 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wssj\" (UniqueName: \"kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj\") pod \"auth-test-model-3dd067-kserve-584557d7df-7dj84\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.498763 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.498671 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:36.824801 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.824770 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:36.874803 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.874257 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84"] Nov 03 16:55:36.876600 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:55:36.874667 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9378757c_40d7_49a5_8926_80d527ce69ad.slice/crio-ff1b77b561e267562c67dc28ed3ec4e0e69b892703a594822455deb9b0bb39eb WatchSource:0}: Error finding container ff1b77b561e267562c67dc28ed3ec4e0e69b892703a594822455deb9b0bb39eb: Status 404 returned error can't find the container with id ff1b77b561e267562c67dc28ed3ec4e0e69b892703a594822455deb9b0bb39eb Nov 03 16:55:36.937202 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.937173 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" event={"ID":"dbf58250-f040-4089-9364-6600e22887ac","Type":"ContainerStarted","Data":"2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5"} Nov 03 16:55:36.938098 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.938077 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:36.939530 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.939470 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" event={"ID":"9378757c-40d7-49a5-8926-80d527ce69ad","Type":"ContainerStarted","Data":"ff1b77b561e267562c67dc28ed3ec4e0e69b892703a594822455deb9b0bb39eb"} Nov 03 16:55:36.978836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:36.978719 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" podStartSLOduration=2.649859261 podStartE2EDuration="2.978560918s" totalImagesPullingTime="328.701657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:36.976760606 +0000 UTC m=+2480.670031629" watchObservedRunningTime="2026-11-03 16:55:36.978560918 +0000 UTC m=+2480.671831940" Nov 03 16:55:37.949577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:37.949535 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" event={"ID":"9378757c-40d7-49a5-8926-80d527ce69ad","Type":"ContainerStarted","Data":"a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d"} Nov 03 16:55:37.993370 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:37.993311 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" podStartSLOduration=1.9932970380000001 podStartE2EDuration="1.993297038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:37.990378162 +0000 UTC m=+2481.683649185" watchObservedRunningTime="2026-11-03 16:55:37.993297038 +0000 UTC m=+2481.686568089" Nov 03 16:55:38.494671 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:38.494637 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:38.499127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:38.499106 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:40.147500 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:40.147459 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:40.293545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:40.293516 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:40.447287 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:40.447195 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:42.293648 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:42.293607 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:42.955533 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:42.955504 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 16:55:43.154125 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.154087 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:43.299027 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.298992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:43.895239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.895175 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t"] Nov 03 16:55:43.899696 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.899669 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:43.919734 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.919234 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-1e9d48-serving-cert\"" Nov 03 16:55:43.931316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.931286 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t"] Nov 03 16:55:43.983409 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.983381 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:43.983611 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.983593 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84f2\" (UniqueName: \"kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:43.983757 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.983737 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:43.983833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:43.983806 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.021745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.021714 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:44.085392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.085353 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.085638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.085404 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.085638 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.085524 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.085737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.085714 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84f2\" (UniqueName: \"kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.086216 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.086191 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.088166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.088146 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.088234 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.088182 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.109484 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.109418 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84f2\" (UniqueName: \"kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2\") pod \"maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.176227 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.175605 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:44.216138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.216101 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:44.537897 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:44.537863 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t"] Nov 03 16:55:44.539885 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:55:44.539852 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c982958_c024_4297_a336_c09c6f42480b.slice/crio-d26956aed0f84ab4bcfb7b8b91369093418d514fb8c855f94f9c49980cd6539d WatchSource:0}: Error finding container d26956aed0f84ab4bcfb7b8b91369093418d514fb8c855f94f9c49980cd6539d: Status 404 returned error can't find the container with id d26956aed0f84ab4bcfb7b8b91369093418d514fb8c855f94f9c49980cd6539d Nov 03 16:55:45.037587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:45.037307 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" event={"ID":"5c982958-c024-4297-a336-c09c6f42480b","Type":"ContainerStarted","Data":"a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701"} Nov 03 16:55:45.037587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:45.037384 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:45.037587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:45.037407 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" event={"ID":"5c982958-c024-4297-a336-c09c6f42480b","Type":"ContainerStarted","Data":"d26956aed0f84ab4bcfb7b8b91369093418d514fb8c855f94f9c49980cd6539d"} Nov 03 16:55:45.078774 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:45.078714 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" podStartSLOduration=1.767059364 podStartE2EDuration="2.07869592s" totalImagesPullingTime="311.636556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:45.076234011 +0000 UTC m=+2488.769505033" watchObservedRunningTime="2026-11-03 16:55:45.07869592 +0000 UTC m=+2488.771966944" Nov 03 16:55:45.786635 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:45.786600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:46.499696 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:46.499661 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:46.512194 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:46.512168 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:47.063720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:47.063688 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:55:47.308907 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:47.308876 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:48.947170 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:48.947132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:50.688186 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:50.688149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:51.051129 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:51.051102 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 16:55:52.342251 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:52.342213 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:54.123239 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:54.123202 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:55.702141 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:55.699108 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:56.176343 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:55:56.176304 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:55:57.704161 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:57.704129 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:58.813973 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:58.813937 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:55:59.781964 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:55:59.781927 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:01.729204 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:01.729172 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:03.803386 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:03.803332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:05.651982 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:05.651945 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:07.176969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:07.176929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:08.881841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:08.881788 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:10.839913 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:10.839866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:12.555469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:12.552091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:12.998619 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:12.998541 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:15.003797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.003759 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:15.781381 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.781345 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6"] Nov 03 16:56:15.786226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.786196 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:15.799587 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.799544 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2b49df\"/\"openshift-service-ca.crt\"" Nov 03 16:56:15.820715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.820687 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2b49df\"/\"default-dockercfg-zptzq\"" Nov 03 16:56:15.820949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.820813 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2b49df\"/\"auth-test-model-2b49df-kserve-self-signed-certs\"" Nov 03 16:56:15.827985 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.827955 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6"] Nov 03 16:56:15.879221 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.879184 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:15.899709 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.899679 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2b49df\"/\"kube-root-ca.crt\"" Nov 03 16:56:15.921379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.921343 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:15.921628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.921397 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhmc\" (UniqueName: \"kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:15.921628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.921505 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:15.921628 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.921560 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:15.921798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.921649 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:15.921798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:15.921717 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.023886 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.023840 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.024515 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.024689 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.024781 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhmc\" (UniqueName: \"kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.024875 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.024936 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.025459 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.025725 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.026362 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.025956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.034114 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.034039 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.043017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.042990 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.062050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.062017 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhmc\" (UniqueName: \"kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc\") pod \"auth-test-model-2b49df-kserve-76b56758d9-nsrl6\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.123327 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.123299 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:16.570519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.570488 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6"] Nov 03 16:56:16.571738 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:56:16.571714 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9388c726_b496_47ff_ba87_67ee3d5e3825.slice/crio-d4acac9a345c239a3768f850384e20f0b75bec3cf0c2ac60c25bf3194cc82e43 WatchSource:0}: Error finding container d4acac9a345c239a3768f850384e20f0b75bec3cf0c2ac60c25bf3194cc82e43: Status 404 returned error can't find the container with id d4acac9a345c239a3768f850384e20f0b75bec3cf0c2ac60c25bf3194cc82e43 Nov 03 16:56:16.859967 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:16.859874 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:17.372745 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:17.372704 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" event={"ID":"9388c726-b496-47ff-ba87-67ee3d5e3825","Type":"ContainerStarted","Data":"03b8b1bd217aacc3e8135d735830915b4492406d9de3e6d7283cc7141d128178"} Nov 03 16:56:17.373250 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:17.372755 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" event={"ID":"9388c726-b496-47ff-ba87-67ee3d5e3825","Type":"ContainerStarted","Data":"d4acac9a345c239a3768f850384e20f0b75bec3cf0c2ac60c25bf3194cc82e43"} Nov 03 16:56:17.424309 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:17.424250 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" podStartSLOduration=2.424236328 podStartE2EDuration="2.424236328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:17.420900865 +0000 UTC m=+2521.114171887" watchObservedRunningTime="2026-11-03 16:56:17.424236328 +0000 UTC m=+2521.117507373" Nov 03 16:56:17.998379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:17.998347 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4"] Nov 03 16:56:17.998611 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:17.998585 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" containerID="cri-o://1875d1f042151c96090c809e2ed9bb07d38695de8fc6fda5ea8e3c933e028026" gracePeriod=30 Nov 03 16:56:18.123948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:18.123901 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:18.470338 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:18.470290 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:20.067466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:20.067410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:21.845972 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:21.845932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:22.983652 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:22.983609 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:56:22.984207 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:22.983683 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:56:23.007927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.007890 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:23.452482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.452414 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:23.476892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.476750 4539 generic.go:356] "Generic (PLEG): container finished" podID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerID="1875d1f042151c96090c809e2ed9bb07d38695de8fc6fda5ea8e3c933e028026" exitCode=0 Nov 03 16:56:23.476892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.476810 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" event={"ID":"d10e1c18-810f-4ee0-affb-934923f0a9d9","Type":"ContainerDied","Data":"1875d1f042151c96090c809e2ed9bb07d38695de8fc6fda5ea8e3c933e028026"} Nov 03 16:56:23.528905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.528876 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:56:23.609135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609041 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609135 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609095 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609142 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609185 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609214 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609253 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609368 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609356 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609376 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609437 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy\") pod \"d10e1c18-810f-4ee0-affb-934923f0a9d9\" (UID: \"d10e1c18-810f-4ee0-affb-934923f0a9d9\") " Nov 03 16:56:23.609660 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609446 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:23.609823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.609799 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.610059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.610032 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:23.610640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.610396 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:23.610640 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.610335 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:56:23.611292 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.610618 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:23.617896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.617872 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:56:23.618013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.617916 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "kube-api-access-6r25x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:56:23.618013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.617952 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:56:23.619545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.619520 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo" pod "d10e1c18-810f-4ee0-affb-934923f0a9d9" (UID: "d10e1c18-810f-4ee0-affb-934923f0a9d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:56:23.711792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711752 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.711792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711787 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.711792 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711797 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.712035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711806 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.712035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711814 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10e1c18-810f-4ee0-affb-934923f0a9d9-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.712035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711822 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10e1c18-810f-4ee0-affb-934923f0a9d9-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.712035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711830 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r25x\" (UniqueName: \"kubernetes.io/projected/d10e1c18-810f-4ee0-affb-934923f0a9d9-kube-api-access-6r25x\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:23.712035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:23.711841 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10e1c18-810f-4ee0-affb-934923f0a9d9-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:56:24.501172 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:24.501141 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" Nov 03 16:56:24.501711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:24.501143 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4" event={"ID":"d10e1c18-810f-4ee0-affb-934923f0a9d9","Type":"ContainerDied","Data":"b2342e5fd1130c7de0c8b347397e4d134efefe9751c868627a1221a6012a9a77"} Nov 03 16:56:24.501711 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:24.501319 4539 scope.go:122] "RemoveContainer" containerID="1875d1f042151c96090c809e2ed9bb07d38695de8fc6fda5ea8e3c933e028026" Nov 03 16:56:24.572447 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:24.572400 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4"] Nov 03 16:56:24.579018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:24.577965 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-g7kl4"] Nov 03 16:56:25.072727 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:25.072684 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:26.124090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:26.124056 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:26.138678 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:26.138649 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:26.159364 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:56:26.159330 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:56:26.470103 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:26.470027 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" path="/var/lib/kubelet/pods/d10e1c18-810f-4ee0-affb-934923f0a9d9/volumes" Nov 03 16:56:26.530675 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:26.530647 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 16:56:26.554488 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:26.554454 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:28.289174 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:28.289140 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:29.869316 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:29.869281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:31.886439 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:31.886393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:33.060883 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:33.060841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:34.692270 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:34.689945 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:35.390297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.390260 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd"] Nov 03 16:56:35.390892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.390875 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" Nov 03 16:56:35.390892 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.390890 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" Nov 03 16:56:35.391090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.391012 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10e1c18-810f-4ee0-affb-934923f0a9d9" containerName="istio-proxy" Nov 03 16:56:35.396050 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.396027 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.417984 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.417947 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd"] Nov 03 16:56:35.421177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.421150 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-aaea00\"/\"kube-root-ca.crt\"" Nov 03 16:56:35.421177 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.421156 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-aaea00\"/\"openshift-service-ca.crt\"" Nov 03 16:56:35.421517 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.421497 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-aaea00\"/\"default-dockercfg-wjkzs\"" Nov 03 16:56:35.421601 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.421502 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-aaea00\"/\"explicit-model-aaea00-kserve-self-signed-certs\"" Nov 03 16:56:35.476654 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.476620 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:35.486704 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.486670 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.486926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.486820 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.486926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.486888 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.487055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.486937 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmw8h\" (UniqueName: \"kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.487055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.486976 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.487124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.487094 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588224 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588649 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588606 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588675 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588706 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmw8h\" (UniqueName: \"kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588777 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588735 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588845 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.588969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588878 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.589071 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.588703 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.589258 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.589238 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.591982 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.591959 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.594245 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.594220 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.613008 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.612981 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmw8h\" (UniqueName: \"kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h\") pod \"explicit-model-aaea00-kserve-6fbc457dbc-75pfd\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:35.723502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:35.722700 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:36.172779 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:36.172749 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd"] Nov 03 16:56:36.173260 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:56:36.173235 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6885ec8_1eb7_4bff_b543_bc031258ef89.slice/crio-8a80bcee39d301ab79447985312881b164179631a164d30ecbef39c73ce8fbcc WatchSource:0}: Error finding container 8a80bcee39d301ab79447985312881b164179631a164d30ecbef39c73ce8fbcc: Status 404 returned error can't find the container with id 8a80bcee39d301ab79447985312881b164179631a164d30ecbef39c73ce8fbcc Nov 03 16:56:36.492200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:36.492105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:36.678552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:36.678042 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" event={"ID":"f6885ec8-1eb7-4bff-b543-bc031258ef89","Type":"ContainerStarted","Data":"7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106"} Nov 03 16:56:36.678552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:36.678096 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" event={"ID":"f6885ec8-1eb7-4bff-b543-bc031258ef89","Type":"ContainerStarted","Data":"8a80bcee39d301ab79447985312881b164179631a164d30ecbef39c73ce8fbcc"} Nov 03 16:56:36.723552 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:36.723516 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:36.726175 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:36.726119 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" podStartSLOduration=1.726099798 podStartE2EDuration="1.726099798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:36.720098292 +0000 UTC m=+2540.413369316" watchObservedRunningTime="2026-11-03 16:56:36.726099798 +0000 UTC m=+2540.419370822" Nov 03 16:56:38.172395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:38.172367 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:39.734408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:39.734372 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:41.386610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:41.383002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:42.695884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:42.695826 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:43.236395 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:43.236320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:44.902124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:44.902083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:45.724603 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:45.724569 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:45.737610 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:45.737583 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:45.759740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:45.759712 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:56:46.446133 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.446098 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b"] Nov 03 16:56:46.451332 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.451309 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.476380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.476217 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"auth-test-model-c6f303-kserve-self-signed-certs\"" Nov 03 16:56:46.476380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.476270 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"kube-root-ca.crt\"" Nov 03 16:56:46.476380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.476316 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"openshift-service-ca.crt\"" Nov 03 16:56:46.476380 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.476351 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"default-dockercfg-66t2h\"" Nov 03 16:56:46.480684 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.477721 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b"] Nov 03 16:56:46.518849 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.518807 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.519104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.518877 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.519104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.518966 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bfb\" (UniqueName: \"kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.519104 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.519031 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.519349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.519150 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.519349 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.519217 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.542508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.542408 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:46.620795 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.620757 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621018 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.620846 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.621021 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.621054 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621092 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.621074 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bfb\" (UniqueName: \"kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621256 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.621110 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.621640 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.621873 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.621710 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.622060 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.622039 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.624054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.624032 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.624335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.624315 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.649954 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.649928 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bfb\" (UniqueName: \"kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb\") pod \"auth-test-model-c6f303-kserve-6779479b77-x466b\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:46.715508 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.708389 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:46.794454 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:46.792367 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:47.172228 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:47.172202 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b"] Nov 03 16:56:47.173102 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:56:47.173075 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafbb4939_8cf4_4751_9128_018f943962f2.slice/crio-6106056753fdcc92b6d6270e116645466adf2faf9afe88c5db23312e94a3fa23 WatchSource:0}: Error finding container 6106056753fdcc92b6d6270e116645466adf2faf9afe88c5db23312e94a3fa23: Status 404 returned error can't find the container with id 6106056753fdcc92b6d6270e116645466adf2faf9afe88c5db23312e94a3fa23 Nov 03 16:56:47.785710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:47.785674 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" event={"ID":"afbb4939-8cf4-4751-9128-018f943962f2","Type":"ContainerStarted","Data":"04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37"} Nov 03 16:56:47.785710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:47.785725 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" event={"ID":"afbb4939-8cf4-4751-9128-018f943962f2","Type":"ContainerStarted","Data":"6106056753fdcc92b6d6270e116645466adf2faf9afe88c5db23312e94a3fa23"} Nov 03 16:56:47.797927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:47.797902 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:47.824577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:47.824532 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" podStartSLOduration=1.824519013 podStartE2EDuration="1.824519013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:47.824223898 +0000 UTC m=+2551.517494921" watchObservedRunningTime="2026-11-03 16:56:47.824519013 +0000 UTC m=+2551.517790036" Nov 03 16:56:48.370744 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:48.370705 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:49.967307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:49.967272 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:51.483048 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:51.483010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:52.482193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:52.482155 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd"] Nov 03 16:56:52.556190 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:52.556158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:52.911605 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:52.911546 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:53.299669 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:53.299639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:54.888529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:54.888411 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:56.157909 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:56:56.157866 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:56:56.416376 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:56.416281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:56.797814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:56.797776 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:56.811920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:56.811883 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:56.872575 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:56.872545 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 16:56:57.930567 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:57.930533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:58.081830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:58.081797 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:56:59.630597 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:56:59.630372 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:00.328598 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:00.328568 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:01.450099 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:01.450049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:03.455086 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:03.455049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:05.372343 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:05.372295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:07.060482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.059974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:07.632301 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.632249 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" podUID="f6885ec8-1eb7-4bff-b543-bc031258ef89" containerName="main" containerID="cri-o://7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106" gracePeriod=45 Nov 03 16:57:07.748833 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.748796 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" podUID="f6885ec8-1eb7-4bff-b543-bc031258ef89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": dial tcp 10.133.0.141:8000: connect: connection refused" Nov 03 16:57:07.921013 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.920979 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:57:07.954811 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.954778 4539 generic.go:356] "Generic (PLEG): container finished" podID="f6885ec8-1eb7-4bff-b543-bc031258ef89" containerID="7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106" exitCode=0 Nov 03 16:57:07.954997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.954830 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" Nov 03 16:57:07.954997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.954857 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" event={"ID":"f6885ec8-1eb7-4bff-b543-bc031258ef89","Type":"ContainerDied","Data":"7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106"} Nov 03 16:57:07.954997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.954898 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd" event={"ID":"f6885ec8-1eb7-4bff-b543-bc031258ef89","Type":"ContainerDied","Data":"8a80bcee39d301ab79447985312881b164179631a164d30ecbef39c73ce8fbcc"} Nov 03 16:57:07.954997 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.954916 4539 scope.go:122] "RemoveContainer" containerID="7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106" Nov 03 16:57:07.967780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.967752 4539 scope.go:122] "RemoveContainer" containerID="7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106" Nov 03 16:57:07.968227 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:57:07.968192 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106\": container with ID starting with 7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106 not found: ID does not exist" containerID="7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106" Nov 03 16:57:07.968320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:07.968238 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106"} err="failed to get container status \"7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106\": rpc error: code = NotFound desc = could not find container \"7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106\": container with ID starting with 7b2ce1bdb3b8ec83263a828194adc094e759baf870b36f9a80bd323bf06b7106 not found: ID does not exist" Nov 03 16:57:08.081957 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.081918 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home\") pod \"f6885ec8-1eb7-4bff-b543-bc031258ef89\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " Nov 03 16:57:08.081957 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.081963 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs\") pod \"f6885ec8-1eb7-4bff-b543-bc031258ef89\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082026 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir\") pod \"f6885ec8-1eb7-4bff-b543-bc031258ef89\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082103 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache\") pod \"f6885ec8-1eb7-4bff-b543-bc031258ef89\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082158 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h\" (UniqueName: \"kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h\") pod \"f6885ec8-1eb7-4bff-b543-bc031258ef89\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082220 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home" pod "f6885ec8-1eb7-4bff-b543-bc031258ef89" (UID: "f6885ec8-1eb7-4bff-b543-bc031258ef89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082233 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm\") pod \"f6885ec8-1eb7-4bff-b543-bc031258ef89\" (UID: \"f6885ec8-1eb7-4bff-b543-bc031258ef89\") " Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082470 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir" pod "f6885ec8-1eb7-4bff-b543-bc031258ef89" (UID: "f6885ec8-1eb7-4bff-b543-bc031258ef89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:08.082576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082512 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache" pod "f6885ec8-1eb7-4bff-b543-bc031258ef89" (UID: "f6885ec8-1eb7-4bff-b543-bc031258ef89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:08.082927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082911 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:57:08.083142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082931 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:57:08.083142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.082955 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:57:08.084749 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.084718 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm" pod "f6885ec8-1eb7-4bff-b543-bc031258ef89" (UID: "f6885ec8-1eb7-4bff-b543-bc031258ef89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:08.084891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.084819 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs" pod "f6885ec8-1eb7-4bff-b543-bc031258ef89" (UID: "f6885ec8-1eb7-4bff-b543-bc031258ef89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:08.085031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.085008 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h" pod "f6885ec8-1eb7-4bff-b543-bc031258ef89" (UID: "f6885ec8-1eb7-4bff-b543-bc031258ef89"). InnerVolumeSpecName "kube-api-access-jmw8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:08.184834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.184740 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6885ec8-1eb7-4bff-b543-bc031258ef89-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:57:08.184834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.184770 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmw8h\" (UniqueName: \"kubernetes.io/projected/f6885ec8-1eb7-4bff-b543-bc031258ef89-kube-api-access-jmw8h\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:57:08.184834 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.184780 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6885ec8-1eb7-4bff-b543-bc031258ef89-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:57:08.299585 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.299551 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd"] Nov 03 16:57:08.305156 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.305125 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-aaea00/explicit-model-aaea00-kserve-6fbc457dbc-75pfd"] Nov 03 16:57:08.471566 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.471486 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6885ec8-1eb7-4bff-b543-bc031258ef89" path="/var/lib/kubelet/pods/f6885ec8-1eb7-4bff-b543-bc031258ef89/volumes" Nov 03 16:57:08.693322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:08.693263 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:10.319127 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:10.319093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:11.985905 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:11.985873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:12.684253 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:12.684194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:13.498293 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:13.498253 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:15.065581 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:15.065547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:16.580740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:16.580703 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:18.137765 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:18.137715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:19.805113 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:19.805081 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:21.493359 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:21.493319 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:23.037688 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:23.037648 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:24.581550 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:24.581502 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:25.942453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.939933 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m"] Nov 03 16:57:25.942453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.940840 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f6885ec8-1eb7-4bff-b543-bc031258ef89" containerName="main" Nov 03 16:57:25.942453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.940860 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6885ec8-1eb7-4bff-b543-bc031258ef89" containerName="main" Nov 03 16:57:25.942453 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.941009 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6885ec8-1eb7-4bff-b543-bc031258ef89" containerName="main" Nov 03 16:57:25.949840 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.949763 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:25.965091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.965062 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m"] Nov 03 16:57:25.970721 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.970682 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"openshift-service-ca.crt\"" Nov 03 16:57:25.970872 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.970762 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"kube-root-ca.crt\"" Nov 03 16:57:25.974720 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.974695 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"auth-test-model-1e9d48-kserve-self-signed-certs\"" Nov 03 16:57:25.975364 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:25.975343 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"default-dockercfg-g9w7v\"" Nov 03 16:57:26.028196 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.028160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:26.079276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.079233 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.079483 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.079365 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.079594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.079530 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.079642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.079610 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.079723 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.079699 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.079776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.079756 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fns\" (UniqueName: \"kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.161803 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:57:26.161754 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:57:26.189679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.188716 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.189679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.188780 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.189679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.188827 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fns\" (UniqueName: \"kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.189679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.188905 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.189679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.188969 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.189679 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.189098 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.190145 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.189990 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.190574 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.190274 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.191069 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.191017 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.205208 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.205098 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.210180 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.209923 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:26.264666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.245158 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.264666 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.245734 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fns\" (UniqueName: \"kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns\") pod \"auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.278449 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.268781 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:26.670580 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:26.670553 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m"] Nov 03 16:57:26.670994 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:57:26.670973 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod878658be_cc2f_4ada_84b0_29798de9a1f0.slice/crio-f90fd818c8a9667d076d3792ff6cb180f6b6591fd97d79e75afbeb4c41bbe9fd WatchSource:0}: Error finding container f90fd818c8a9667d076d3792ff6cb180f6b6591fd97d79e75afbeb4c41bbe9fd: Status 404 returned error can't find the container with id f90fd818c8a9667d076d3792ff6cb180f6b6591fd97d79e75afbeb4c41bbe9fd Nov 03 16:57:27.119109 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:27.119058 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" event={"ID":"878658be-cc2f-4ada-84b0-29798de9a1f0","Type":"ContainerStarted","Data":"1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598"} Nov 03 16:57:27.119609 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:27.119118 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" event={"ID":"878658be-cc2f-4ada-84b0-29798de9a1f0","Type":"ContainerStarted","Data":"f90fd818c8a9667d076d3792ff6cb180f6b6591fd97d79e75afbeb4c41bbe9fd"} Nov 03 16:57:27.161504 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:27.161448 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" podStartSLOduration=2.161413293 podStartE2EDuration="2.161413293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:57:27.157599881 +0000 UTC m=+2590.850870905" watchObservedRunningTime="2026-11-03 16:57:27.161413293 +0000 UTC m=+2590.854684315" Nov 03 16:57:27.279132 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:27.279102 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:27.951403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:27.951341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:29.658263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:29.658234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:31.271031 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:31.270993 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:32.905274 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:32.905241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:34.783812 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:34.783777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:35.613519 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:35.613472 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:35.769959 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:35.769929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:35.922300 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:35.922207 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:36.279017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:36.278988 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:36.292200 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:36.292177 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:36.722114 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:36.721969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:37.217558 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:37.217531 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:57:38.350496 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:38.350457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:39.884396 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:39.884364 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:41.410253 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:41.390866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:41.589448 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:41.583342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:42.272211 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:42.272165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:43.121798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:43.121744 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:44.767372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:44.767318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:46.523173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:46.523107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:47.772934 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:47.772890 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m"] Nov 03 16:57:47.852714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:47.852666 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:48.472413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:48.472382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:48.636639 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:48.636604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:49.171697 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:49.171660 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:49.510356 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:49.510254 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:49.713105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:49.713067 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:51.412224 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:51.412190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:53.189173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:53.189127 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:53.501487 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:53.501366 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:53.684056 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:53.684024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:54.799313 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:54.799269 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:55.127518 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:55.127408 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:55.291262 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:55.291227 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:56.158014 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:57:56.157974 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:57:56.773903 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:56.773867 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:58.007348 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:58.007305 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:58.159718 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:58.159683 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:58.397112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:58.397078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:58.815373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:58.815324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:59.224722 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:59.224630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:57:59.867865 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:57:59.867814 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:00.311684 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:00.311640 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:00.512462 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:00.503453 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:00.779796 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:00.779761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:01.031949 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:01.031865 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd"] Nov 03 16:58:01.042472 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:01.042394 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da878e9-wp6xd"] Nov 03 16:58:02.080495 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:02.080457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:02.470111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:02.470034 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e09a396-99f5-40a0-899c-cba468ab1655" path="/var/lib/kubelet/pods/4e09a396-99f5-40a0-899c-cba468ab1655/volumes" Nov 03 16:58:02.480687 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:02.480640 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:02.682011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:02.681982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:02.826460 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:02.826400 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" podUID="878658be-cc2f-4ada-84b0-29798de9a1f0" containerName="main" containerID="cri-o://1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598" gracePeriod=45 Nov 03 16:58:02.871961 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:02.871935 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:03.166747 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.166714 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:58:03.290265 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290234 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns\" (UniqueName: \"kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns\") pod \"878658be-cc2f-4ada-84b0-29798de9a1f0\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " Nov 03 16:58:03.290464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290286 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir\") pod \"878658be-cc2f-4ada-84b0-29798de9a1f0\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " Nov 03 16:58:03.290464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290367 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home\") pod \"878658be-cc2f-4ada-84b0-29798de9a1f0\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " Nov 03 16:58:03.290594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290485 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache\") pod \"878658be-cc2f-4ada-84b0-29798de9a1f0\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " Nov 03 16:58:03.290594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290527 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm\") pod \"878658be-cc2f-4ada-84b0-29798de9a1f0\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " Nov 03 16:58:03.290594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290565 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs\") pod \"878658be-cc2f-4ada-84b0-29798de9a1f0\" (UID: \"878658be-cc2f-4ada-84b0-29798de9a1f0\") " Nov 03 16:58:03.290758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290616 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir" pod "878658be-cc2f-4ada-84b0-29798de9a1f0" (UID: "878658be-cc2f-4ada-84b0-29798de9a1f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:03.290758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290685 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home" pod "878658be-cc2f-4ada-84b0-29798de9a1f0" (UID: "878658be-cc2f-4ada-84b0-29798de9a1f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:03.290830 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.290756 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache" pod "878658be-cc2f-4ada-84b0-29798de9a1f0" (UID: "878658be-cc2f-4ada-84b0-29798de9a1f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:03.291142 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.291122 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:03.291413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.291396 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:03.291594 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.291563 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:03.296823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.296794 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm" pod "878658be-cc2f-4ada-84b0-29798de9a1f0" (UID: "878658be-cc2f-4ada-84b0-29798de9a1f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:03.296931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.296819 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns" pod "878658be-cc2f-4ada-84b0-29798de9a1f0" (UID: "878658be-cc2f-4ada-84b0-29798de9a1f0"). InnerVolumeSpecName "kube-api-access-j7fns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:03.296931 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.296833 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs" pod "878658be-cc2f-4ada-84b0-29798de9a1f0" (UID: "878658be-cc2f-4ada-84b0-29798de9a1f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:58:03.393242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.393165 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fns\" (UniqueName: \"kubernetes.io/projected/878658be-cc2f-4ada-84b0-29798de9a1f0-kube-api-access-j7fns\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:03.393242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.393193 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878658be-cc2f-4ada-84b0-29798de9a1f0-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:03.393242 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.393203 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878658be-cc2f-4ada-84b0-29798de9a1f0-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:03.573107 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.573074 4539 generic.go:356] "Generic (PLEG): container finished" podID="878658be-cc2f-4ada-84b0-29798de9a1f0" containerID="1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598" exitCode=0 Nov 03 16:58:03.573276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.573126 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" Nov 03 16:58:03.573276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.573168 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" event={"ID":"878658be-cc2f-4ada-84b0-29798de9a1f0","Type":"ContainerDied","Data":"1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598"} Nov 03 16:58:03.573276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.573221 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m" event={"ID":"878658be-cc2f-4ada-84b0-29798de9a1f0","Type":"ContainerDied","Data":"f90fd818c8a9667d076d3792ff6cb180f6b6591fd97d79e75afbeb4c41bbe9fd"} Nov 03 16:58:03.573276 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.573247 4539 scope.go:122] "RemoveContainer" containerID="1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598" Nov 03 16:58:03.592244 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.592217 4539 scope.go:122] "RemoveContainer" containerID="1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598" Nov 03 16:58:03.592604 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:58:03.592578 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598\": container with ID starting with 1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598 not found: ID does not exist" containerID="1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598" Nov 03 16:58:03.592703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.592614 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598"} err="failed to get container status \"1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598\": rpc error: code = NotFound desc = could not find container \"1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598\": container with ID starting with 1d2c2b1ce182b2034d0f16dd7bd3d061a3f0180446a70102e77dcd236f60a598 not found: ID does not exist" Nov 03 16:58:03.636117 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.636082 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m"] Nov 03 16:58:03.638737 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.638710 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/auth-test-model-1e9d48-kserve-76d67f84d5-s7v2m"] Nov 03 16:58:03.993318 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:03.993281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:04.469714 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:04.469684 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878658be-cc2f-4ada-84b0-29798de9a1f0" path="/var/lib/kubelet/pods/878658be-cc2f-4ada-84b0-29798de9a1f0/volumes" Nov 03 16:58:05.645059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:05.645026 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:07.329267 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:07.329075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:08.888288 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:08.888255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:10.582944 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:10.582907 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:12.030596 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:12.030561 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m"] Nov 03 16:58:12.040857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:12.040826 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d78bebd-jv48m"] Nov 03 16:58:12.288481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:12.288358 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:12.470087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:12.470054 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1514118-889d-479c-ad17-9a7f322e086e" path="/var/lib/kubelet/pods/b1514118-889d-479c-ad17-9a7f322e086e/volumes" Nov 03 16:58:12.699464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:12.699393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:12.892325 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:12.892238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:13.410258 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:13.410219 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629"] Nov 03 16:58:13.410743 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:13.410528 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" containerID="cri-o://b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2" gracePeriod=30 Nov 03 16:58:14.081089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:14.081047 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:15.391798 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:15.391764 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:15.663764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:15.663675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:15.896163 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:15.896124 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tw629 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:58:15.896344 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:15.896193 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:58:17.304360 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.304323 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z"] Nov 03 16:58:17.304969 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.304953 4539 cpu_manager.go:414] "containerMap: removing container" podUID="878658be-cc2f-4ada-84b0-29798de9a1f0" containerName="main" Nov 03 16:58:17.305025 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.304971 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878658be-cc2f-4ada-84b0-29798de9a1f0" containerName="main" Nov 03 16:58:17.305093 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.305084 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="878658be-cc2f-4ada-84b0-29798de9a1f0" containerName="main" Nov 03 16:58:17.308645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.308628 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.326095 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.326070 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c1837e-serving-cert\"" Nov 03 16:58:17.344347 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.344312 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z"] Nov 03 16:58:17.419096 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.419056 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:17.433853 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.433813 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwkf\" (UniqueName: \"kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.434006 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.433967 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.434053 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.434024 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.434143 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.434131 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.536165 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.536113 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwkf\" (UniqueName: \"kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.536372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.536232 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.536372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.536290 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.536372 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.536352 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.537994 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.537956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.541054 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.540835 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.544166 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.544140 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.559764 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.559700 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwkf\" (UniqueName: \"kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf\") pod \"maas-api-e2e-autores-c1837e-749779ccc7-4zq5z\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.561729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.561700 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:17.663469 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.663436 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:17.976961 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.976933 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z"] Nov 03 16:58:17.977920 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:58:17.977886 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1987c4e0_a459_4fdf_9489_c7943b68314e.slice/crio-d6540993ff85cd2570b80087d975b898c1e3f029e33765c5111a0b231c47da6e WatchSource:0}: Error finding container d6540993ff85cd2570b80087d975b898c1e3f029e33765c5111a0b231c47da6e: Status 404 returned error can't find the container with id d6540993ff85cd2570b80087d975b898c1e3f029e33765c5111a0b231c47da6e Nov 03 16:58:17.979417 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:17.979399 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:58:18.740964 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.740766 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" event={"ID":"1987c4e0-a459-4fdf-9489-c7943b68314e","Type":"ContainerStarted","Data":"1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e"} Nov 03 16:58:18.740964 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.740826 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" event={"ID":"1987c4e0-a459-4fdf-9489-c7943b68314e","Type":"ContainerStarted","Data":"d6540993ff85cd2570b80087d975b898c1e3f029e33765c5111a0b231c47da6e"} Nov 03 16:58:18.740964 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.740854 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:18.742896 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.742874 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:58:18.743007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.742921 4539 generic.go:356] "Generic (PLEG): container finished" podID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerID="b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2" exitCode=0 Nov 03 16:58:18.743007 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.742965 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" event={"ID":"9eaf3234-d440-4b4d-874b-28235bcb8cac","Type":"ContainerDied","Data":"b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2"} Nov 03 16:58:18.743100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.743013 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" event={"ID":"9eaf3234-d440-4b4d-874b-28235bcb8cac","Type":"ContainerDied","Data":"a0a0590522e6b4b601766eb02e6f8922c400c042f80821081462a1ca492cd325"} Nov 03 16:58:18.743100 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.743037 4539 scope.go:122] "RemoveContainer" containerID="b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2" Nov 03 16:58:18.746670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.746645 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.746818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.746709 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.746818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.746746 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.746818 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.746790 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.747035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.746877 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.747035 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.746927 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.747164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747038 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:18.747164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747083 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:18.747164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747103 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.747164 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747137 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.747912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747225 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:18.747912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747259 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp\") pod \"9eaf3234-d440-4b4d-874b-28235bcb8cac\" (UID: \"9eaf3234-d440-4b4d-874b-28235bcb8cac\") " Nov 03 16:58:18.747912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747363 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:18.747912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747462 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:18.748168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.747988 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf3234-d440-4b4d-874b-28235bcb8cac-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.748168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.748009 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.748168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.748025 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.748168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.748040 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.748168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.748095 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.750780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.750752 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "kube-api-access-m2sbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:18.751017 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.750991 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:18.751118 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.751097 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:58:18.751393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.751369 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy" pod "9eaf3234-d440-4b4d-874b-28235bcb8cac" (UID: "9eaf3234-d440-4b4d-874b-28235bcb8cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:18.757470 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.757453 4539 scope.go:122] "RemoveContainer" containerID="b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2" Nov 03 16:58:18.757811 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:58:18.757784 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2\": container with ID starting with b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2 not found: ID does not exist" containerID="b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2" Nov 03 16:58:18.757912 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.757820 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2"} err="failed to get container status \"b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2\": rpc error: code = NotFound desc = could not find container \"b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2\": container with ID starting with b86df9a9249f05a8e4a76f40044ece874a412930211aec58bf28c4fe161814d2 not found: ID does not exist" Nov 03 16:58:18.782335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.782285 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" podStartSLOduration=1.292208249 podStartE2EDuration="1.782269387s" totalImagesPullingTime="490.061138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:58:18.779875244 +0000 UTC m=+2642.473146267" watchObservedRunningTime="2026-11-03 16:58:18.782269387 +0000 UTC m=+2642.475540409" Nov 03 16:58:18.850228 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.850185 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.850228 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.850225 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.850534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.850241 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2sbp\" (UniqueName: \"kubernetes.io/projected/9eaf3234-d440-4b4d-874b-28235bcb8cac-kube-api-access-m2sbp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:18.850534 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:18.850258 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf3234-d440-4b4d-874b-28235bcb8cac-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:58:19.064870 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:19.064842 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:19.752173 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:19.752146 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629" Nov 03 16:58:19.801457 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:19.801412 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629"] Nov 03 16:58:19.813664 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:19.813630 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tw629"] Nov 03 16:58:20.472191 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:20.472155 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" path="/var/lib/kubelet/pods/9eaf3234-d440-4b4d-874b-28235bcb8cac/volumes" Nov 03 16:58:20.681379 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:20.680203 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:22.315105 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:22.315062 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:23.984408 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:23.983913 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:24.758392 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:24.758363 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 16:58:25.481226 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:25.481186 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:26.158277 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:58:26.158237 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:58:27.054829 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:27.054787 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:27.457111 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:27.457011 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:27.751501 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:27.750899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:27.998642 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:27.998611 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:28.181692 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:28.181659 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:28.761946 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:28.761902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:29.047077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:29.047044 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7"] Nov 03 16:58:29.057275 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:29.057238 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1be90c-p8xn7"] Nov 03 16:58:30.379090 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:30.379061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:30.470960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:30.470916 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d21824-fbd7-4524-93a9-1275670dc513" path="/var/lib/kubelet/pods/d4d21824-fbd7-4524-93a9-1275670dc513/volumes" Nov 03 16:58:30.625201 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:30.625165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:30.771536 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:30.771396 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:31.680403 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:31.680372 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:31.821703 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:31.821667 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:31.976493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:31.976376 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:33.197947 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:33.197914 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:33.922819 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:33.922790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:35.552335 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:35.552306 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:37.188353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:37.188315 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:38.781232 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:38.781187 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:40.360278 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:40.360099 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:42.085740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:42.085653 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:43.780498 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:43.780461 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:44.065670 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:44.065579 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq"] Nov 03 16:58:44.074979 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:44.074955 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca14a94-4z5pq"] Nov 03 16:58:44.470291 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:44.470213 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9c1086-d742-4cce-b02a-ed514d30cb72" path="/var/lib/kubelet/pods/5c9c1086-d742-4cce-b02a-ed514d30cb72/volumes" Nov 03 16:58:45.450989 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:45.450950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:47.017827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:47.017789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:48.600451 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:48.597882 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:50.181810 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:50.181772 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:51.368820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:51.368773 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:52.026043 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:52.026005 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:53.604729 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:53.604664 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:55.211455 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:55.208950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:56.158139 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:58:56.158101 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:58:56.800715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:56.800672 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:58.372078 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:58.372042 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:58:59.946593 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:58:59.946558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:01.868281 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:01.868237 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:03.505248 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:03.505214 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:05.505220 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:05.505184 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:07.585780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:07.585748 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:09.252869 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:09.252838 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:09.650913 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:09.650877 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:10.971932 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:10.971879 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:11.309464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.309414 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:11.526464 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.526415 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11342-752024"] Nov 03 16:59:11.527314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.527283 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" Nov 03 16:59:11.527314 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.527309 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" Nov 03 16:59:11.527562 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.527536 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eaf3234-d440-4b4d-874b-28235bcb8cac" containerName="istio-proxy" Nov 03 16:59:11.531836 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.531813 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:11.552097 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.552066 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11342-752024"] Nov 03 16:59:11.556951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.556873 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 16:59:11.634042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.633956 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcvm\" (UniqueName: \"kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm\") pod \"test-curl-11342-752024\" (UID: \"ca64c99b-95e9-4765-89b5-b021b710f4c3\") " pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:11.735539 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.735499 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcvm\" (UniqueName: \"kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm\") pod \"test-curl-11342-752024\" (UID: \"ca64c99b-95e9-4765-89b5-b021b710f4c3\") " pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:11.763158 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.763128 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcvm\" (UniqueName: \"kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm\") pod \"test-curl-11342-752024\" (UID: \"ca64c99b-95e9-4765-89b5-b021b710f4c3\") " pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:11.845884 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:11.845849 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:12.166068 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:59:12.166034 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca64c99b_95e9_4765_89b5_b021b710f4c3.slice/crio-6c1e02d00f42932b9a7f41ad7c059ce7660707fad0168f1a76d2d7bafaf7702e WatchSource:0}: Error finding container 6c1e02d00f42932b9a7f41ad7c059ce7660707fad0168f1a76d2d7bafaf7702e: Status 404 returned error can't find the container with id 6c1e02d00f42932b9a7f41ad7c059ce7660707fad0168f1a76d2d7bafaf7702e Nov 03 16:59:12.166474 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:12.166252 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11342-752024"] Nov 03 16:59:12.193042 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:12.193011 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11342-752024" event={"ID":"ca64c99b-95e9-4765-89b5-b021b710f4c3","Type":"ContainerStarted","Data":"6c1e02d00f42932b9a7f41ad7c059ce7660707fad0168f1a76d2d7bafaf7702e"} Nov 03 16:59:12.743852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:12.743819 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:12.997032 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:12.996949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:13.201363 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:13.201318 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11342-752024" event={"ID":"ca64c99b-95e9-4765-89b5-b021b710f4c3","Type":"ContainerStarted","Data":"30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0"} Nov 03 16:59:13.244556 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:13.244493 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11342-752024" podStartSLOduration=2.244475167 podStartE2EDuration="2.244475167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:13.243532912 +0000 UTC m=+2696.936803934" watchObservedRunningTime="2026-11-03 16:59:13.244475167 +0000 UTC m=+2696.937746190" Nov 03 16:59:14.401486 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:14.401451 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11342-752024"] Nov 03 16:59:14.407116 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:14.407087 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11342-752024"] Nov 03 16:59:14.676389 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:14.675605 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:15.043107 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:15.043073 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:15.218070 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:15.218034 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11342-752024" podUID="ca64c99b-95e9-4765-89b5-b021b710f4c3" containerName="test-curl-11342-752024" containerID="cri-o://30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0" gracePeriod=2 Nov 03 16:59:15.639710 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:15.639669 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:15.876342 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:15.876282 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:16.471855 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:16.471811 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:16.995831 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:16.995800 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:59:17.005045 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.005023 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 16:59:17.494960 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.494920 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:17.534890 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.534858 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm\" (UniqueName: \"kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm\") pod \"ca64c99b-95e9-4765-89b5-b021b710f4c3\" (UID: \"ca64c99b-95e9-4765-89b5-b021b710f4c3\") " Nov 03 16:59:17.537481 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.537452 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm" pod "ca64c99b-95e9-4765-89b5-b021b710f4c3" (UID: "ca64c99b-95e9-4765-89b5-b021b710f4c3"). InnerVolumeSpecName "kube-api-access-2vcvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:17.636748 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.636712 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vcvm\" (UniqueName: \"kubernetes.io/projected/ca64c99b-95e9-4765-89b5-b021b710f4c3-kube-api-access-2vcvm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:17.713020 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.712984 4539 scope.go:122] "RemoveContainer" containerID="cbfcfba032e3eb5906111802dcdbaa2952503d8bdf2cecdad20d1213f5424d38" Nov 03 16:59:17.726655 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.726624 4539 scope.go:122] "RemoveContainer" containerID="a7cf16bbaf8ff85f6f70b0cca5d78809984fe217f4b4d2557ec317060b26ce11" Nov 03 16:59:17.921613 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:17.921518 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:18.243463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.243333 4539 generic.go:356] "Generic (PLEG): container finished" podID="ca64c99b-95e9-4765-89b5-b021b710f4c3" containerID="30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0" exitCode=137 Nov 03 16:59:18.243463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.243387 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:18.243463 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.243396 4539 scope.go:122] "RemoveContainer" containerID="30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0" Nov 03 16:59:18.255473 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.255454 4539 scope.go:122] "RemoveContainer" containerID="30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0" Nov 03 16:59:18.255784 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:18.255761 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0\": container with ID starting with 30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0 not found: ID does not exist" containerID="30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0" Nov 03 16:59:18.255848 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.255794 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0"} err="failed to get container status \"30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0\": rpc error: code = NotFound desc = could not find container \"30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0\": container with ID starting with 30d0dab853d0ddaf5ea1d0dc9feeb4213278db4e08bd9e76a2724ba6d0644cf0 not found: ID does not exist" Nov 03 16:59:18.263997 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:18.263964 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11342-752024\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="ca64c99b-95e9-4765-89b5-b021b710f4c3" pod="opendatahub/test-curl-11342-752024" Nov 03 16:59:18.319102 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.319068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:18.472880 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.472843 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca64c99b-95e9-4765-89b5-b021b710f4c3" path="/var/lib/kubelet/pods/ca64c99b-95e9-4765-89b5-b021b710f4c3/volumes" Nov 03 16:59:18.496926 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:18.496833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:19.830009 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:19.829970 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:20.000814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:20.000780 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:20.564322 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:20.564269 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:21.101948 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:21.101908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:21.614059 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:21.614011 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:22.022782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:22.022689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:22.744240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:22.744190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:23.453978 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:23.453943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:24.777963 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:24.777931 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:26.163227 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:26.163193 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:59:26.455841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:26.455750 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:28.080077 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:28.076296 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:29.757828 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:29.757795 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:31.579920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:31.579778 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:33.164994 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:33.163579 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:35.216410 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:35.216349 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:36.822482 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:36.820774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:38.649075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:38.649038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:40.230653 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:40.230623 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:40.389162 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:40.389129 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:41.842951 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:41.842905 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:42.054094 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:42.054051 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84"] Nov 03 16:59:42.083832 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:42.083802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:42.239101 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:42.238968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:43.690459 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:43.690379 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:45.408252 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:45.408221 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:45.639168 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:45.639134 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:47.404153 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:47.404123 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:49.205637 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:49.205603 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:49.372641 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:49.372604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:49.637607 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:49.637568 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6"] Nov 03 16:59:49.731770 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:49.731675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:50.884823 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:50.884776 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24"] Nov 03 16:59:50.885320 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:50.885031 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" containerID="cri-o://67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda" gracePeriod=30 Nov 03 16:59:50.895055 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:50.895016 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj"] Nov 03 16:59:50.895297 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:50.895250 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" containerID="cri-o://404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0" gracePeriod=30 Nov 03 16:59:50.898682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:50.898657 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6"] Nov 03 16:59:50.898891 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:50.898855 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" containerID="cri-o://05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e" gracePeriod=30 Nov 03 16:59:51.149988 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:51.149897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:52.843867 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:52.843834 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:54.750437 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:54.750298 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:55.161478 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:55.161438 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g"] Nov 03 16:59:55.161682 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:55.161628 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" containerID="cri-o://a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b" gracePeriod=30 Nov 03 16:59:55.171699 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:55.171667 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6"] Nov 03 16:59:55.171950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:55.171927 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" containerID="cri-o://ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a" gracePeriod=30 Nov 03 16:59:55.175919 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:55.175894 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6"] Nov 03 16:59:55.176087 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:55.176069 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" containerID="cri-o://21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3" gracePeriod=30 Nov 03 16:59:56.058725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.058393 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:59:56.058725 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.058469 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:59:56.191127 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:56.191078 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 16:59:56.245904 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.245874 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:59:56.258257 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.258232 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:59:56.290782 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.290760 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:59:56.383529 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383498 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.383736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383539 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.383736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383568 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.383736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383597 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.383736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383680 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.383736 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383711 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383746 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383801 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383837 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383853 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383896 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383931 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383957 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.384028 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.383992 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384046 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384083 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384122 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384186 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384315 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384362 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384375 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384394 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.384490 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384470 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.384968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384501 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.384968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384741 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.384968 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.384852 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:56.385128 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385000 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.385128 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385084 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.385128 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385089 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.385290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385205 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy\") pod \"9394bf9b-9771-4f6e-97cc-e63802c79795\" (UID: \"9394bf9b-9771-4f6e-97cc-e63802c79795\") " Nov 03 16:59:56.385290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385248 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.385290 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385275 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.385466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385301 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo\") pod \"58cd0726-b63e-432f-818a-0e089054a8e3\" (UID: \"58cd0726-b63e-432f-818a-0e089054a8e3\") " Nov 03 16:59:56.385466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385340 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.385466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385354 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token\") pod \"6d57cdfd-557b-4f81-b661-7b87125e8529\" (UID: \"6d57cdfd-557b-4f81-b661-7b87125e8529\") " Nov 03 16:59:56.385466 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385360 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.385886 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386052 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386072 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386087 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386100 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d57cdfd-557b-4f81-b661-7b87125e8529-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386113 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386126 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386139 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386151 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386164 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd0726-b63e-432f-818a-0e089054a8e3-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.386219 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.386175 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.387228 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.387177 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.387852 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.387825 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.389513 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.389466 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.389715 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.389578 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.389797 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.389747 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:56.389866 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.389840 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:56.390075 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.389970 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "kube-api-access-v97tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:56.390888 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.390864 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "kube-api-access-lf6fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:56.390983 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.390909 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:56.391378 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.391358 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:56.392317 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.392296 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:56.392600 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.392584 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.397758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.397737 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.397758 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.397746 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:56.397882 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.397763 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo" pod "6d57cdfd-557b-4f81-b661-7b87125e8529" (UID: "6d57cdfd-557b-4f81-b661-7b87125e8529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:56.397950 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.397927 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv" pod "58cd0726-b63e-432f-818a-0e089054a8e3" (UID: "58cd0726-b63e-432f-818a-0e089054a8e3"). InnerVolumeSpecName "kube-api-access-ghmlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:56.399493 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.399417 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token" pod "9394bf9b-9771-4f6e-97cc-e63802c79795" (UID: "9394bf9b-9771-4f6e-97cc-e63802c79795"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:56.487307 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487278 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd0726-b63e-432f-818a-0e089054a8e3-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487312 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487328 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487340 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487353 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487365 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd0726-b63e-432f-818a-0e089054a8e3-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487378 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmlv\" (UniqueName: \"kubernetes.io/projected/58cd0726-b63e-432f-818a-0e089054a8e3-kube-api-access-ghmlv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487392 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487406 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487442 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487419 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9394bf9b-9771-4f6e-97cc-e63802c79795-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487450 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf6fw\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-kube-api-access-lf6fw\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487463 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487477 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97tt\" (UniqueName: \"kubernetes.io/projected/6d57cdfd-557b-4f81-b661-7b87125e8529-kube-api-access-v97tt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487490 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487503 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487515 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9394bf9b-9771-4f6e-97cc-e63802c79795-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.487740 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.487527 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d57cdfd-557b-4f81-b661-7b87125e8529-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:56.558029 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.557992 4539 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:59:56.558193 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.558046 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:59:56.634502 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.634409 4539 generic.go:356] "Generic (PLEG): container finished" podID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerID="67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda" exitCode=0 Nov 03 16:59:56.634658 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.634525 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" Nov 03 16:59:56.634733 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.634684 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" event={"ID":"9394bf9b-9771-4f6e-97cc-e63802c79795","Type":"ContainerDied","Data":"67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda"} Nov 03 16:59:56.634773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.634738 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24" event={"ID":"9394bf9b-9771-4f6e-97cc-e63802c79795","Type":"ContainerDied","Data":"b0e520181c67db0b99fabac49cd0a5c791d32fe6ab00f37926955ac5b3781f81"} Nov 03 16:59:56.634820 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.634772 4539 scope.go:122] "RemoveContainer" containerID="67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda" Nov 03 16:59:56.637217 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.637188 4539 generic.go:356] "Generic (PLEG): container finished" podID="58cd0726-b63e-432f-818a-0e089054a8e3" containerID="404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0" exitCode=0 Nov 03 16:59:56.637352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.637307 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" event={"ID":"58cd0726-b63e-432f-818a-0e089054a8e3","Type":"ContainerDied","Data":"404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0"} Nov 03 16:59:56.637352 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.637347 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" event={"ID":"58cd0726-b63e-432f-818a-0e089054a8e3","Type":"ContainerDied","Data":"2a2150d584f995cec388a25f441ee62e5e0a9cbfd1f338a34fb17bbfe44f7cf3"} Nov 03 16:59:56.637548 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.637416 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj" Nov 03 16:59:56.639813 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.639788 4539 generic.go:356] "Generic (PLEG): container finished" podID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerID="05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e" exitCode=0 Nov 03 16:59:56.640545 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.640505 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" event={"ID":"6d57cdfd-557b-4f81-b661-7b87125e8529","Type":"ContainerDied","Data":"05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e"} Nov 03 16:59:56.640663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.640554 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" event={"ID":"6d57cdfd-557b-4f81-b661-7b87125e8529","Type":"ContainerDied","Data":"f5bf0f826219971159ed53a3262e71e19b6489b6532a57dd3a6ddb4c6ca588ff"} Nov 03 16:59:56.640663 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.640620 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6" Nov 03 16:59:56.652218 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.652195 4539 scope.go:122] "RemoveContainer" containerID="67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda" Nov 03 16:59:56.652584 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:56.652556 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda\": container with ID starting with 67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda not found: ID does not exist" containerID="67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda" Nov 03 16:59:56.652667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.652597 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda"} err="failed to get container status \"67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda\": rpc error: code = NotFound desc = could not find container \"67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda\": container with ID starting with 67e2e104b538267836c3d7040f33723f0feb008d9cd004f70bc34d1bf6651eda not found: ID does not exist" Nov 03 16:59:56.652667 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.652623 4539 scope.go:122] "RemoveContainer" containerID="404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0" Nov 03 16:59:56.678298 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.678278 4539 scope.go:122] "RemoveContainer" containerID="404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0" Nov 03 16:59:56.678864 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:56.678839 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0\": container with ID starting with 404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0 not found: ID does not exist" containerID="404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0" Nov 03 16:59:56.678975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.678872 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0"} err="failed to get container status \"404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0\": rpc error: code = NotFound desc = could not find container \"404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0\": container with ID starting with 404f0beeca43846f6dce4fb53088a66c573a70441d5437271c433d471bb270c0 not found: ID does not exist" Nov 03 16:59:56.678975 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.678889 4539 scope.go:122] "RemoveContainer" containerID="05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e" Nov 03 16:59:56.691089 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.691069 4539 scope.go:122] "RemoveContainer" containerID="05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e" Nov 03 16:59:56.691358 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:56.691337 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e\": container with ID starting with 05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e not found: ID does not exist" containerID="05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e" Nov 03 16:59:56.691402 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.691369 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e"} err="failed to get container status \"05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e\": rpc error: code = NotFound desc = could not find container \"05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e\": container with ID starting with 05559e3164e8c625bb5bf312991ce78e805a62e002dbea0bd51957832a61c54e not found: ID does not exist" Nov 03 16:59:56.696491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.696470 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj"] Nov 03 16:59:56.701124 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.701100 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-27cjj"] Nov 03 16:59:56.737952 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.737861 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6"] Nov 03 16:59:56.768393 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.768361 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-hbwf6"] Nov 03 16:59:56.802920 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.802888 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24"] Nov 03 16:59:56.817841 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.817810 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffg24"] Nov 03 16:59:56.895586 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:56.895533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:57.086991 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.086948 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" podUID="9378757c-40d7-49a5-8926-80d527ce69ad" containerName="main" containerID="cri-o://a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d" gracePeriod=45 Nov 03 16:59:57.374776 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.374749 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:59:57.496927 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.496894 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj\" (UniqueName: \"kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj\") pod \"9378757c-40d7-49a5-8926-80d527ce69ad\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " Nov 03 16:59:57.497112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497009 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs\") pod \"9378757c-40d7-49a5-8926-80d527ce69ad\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " Nov 03 16:59:57.497112 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497084 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home\") pod \"9378757c-40d7-49a5-8926-80d527ce69ad\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " Nov 03 16:59:57.497240 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497203 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir\") pod \"9378757c-40d7-49a5-8926-80d527ce69ad\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " Nov 03 16:59:57.497298 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497249 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm\") pod \"9378757c-40d7-49a5-8926-80d527ce69ad\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " Nov 03 16:59:57.497353 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497299 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home" pod "9378757c-40d7-49a5-8926-80d527ce69ad" (UID: "9378757c-40d7-49a5-8926-80d527ce69ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:57.497413 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497370 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache\") pod \"9378757c-40d7-49a5-8926-80d527ce69ad\" (UID: \"9378757c-40d7-49a5-8926-80d527ce69ad\") " Nov 03 16:59:57.497577 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497449 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir" pod "9378757c-40d7-49a5-8926-80d527ce69ad" (UID: "9378757c-40d7-49a5-8926-80d527ce69ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:57.497661 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.497639 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache" pod "9378757c-40d7-49a5-8926-80d527ce69ad" (UID: "9378757c-40d7-49a5-8926-80d527ce69ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:57.498138 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.498115 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.498231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.498143 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.498231 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.498182 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.499576 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.499553 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj" pod "9378757c-40d7-49a5-8926-80d527ce69ad" (UID: "9378757c-40d7-49a5-8926-80d527ce69ad"). InnerVolumeSpecName "kube-api-access-8wssj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:57.499709 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.499683 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs" pod "9378757c-40d7-49a5-8926-80d527ce69ad" (UID: "9378757c-40d7-49a5-8926-80d527ce69ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:57.499814 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.499752 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm" pod "9378757c-40d7-49a5-8926-80d527ce69ad" (UID: "9378757c-40d7-49a5-8926-80d527ce69ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:57.599345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.599310 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378757c-40d7-49a5-8926-80d527ce69ad-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.599345 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.599346 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wssj\" (UniqueName: \"kubernetes.io/projected/9378757c-40d7-49a5-8926-80d527ce69ad-kube-api-access-8wssj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.599617 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.599362 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378757c-40d7-49a5-8926-80d527ce69ad-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.652134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.652043 4539 generic.go:356] "Generic (PLEG): container finished" podID="9378757c-40d7-49a5-8926-80d527ce69ad" containerID="a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d" exitCode=0 Nov 03 16:59:57.652134 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.652088 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" Nov 03 16:59:57.652373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.652131 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" event={"ID":"9378757c-40d7-49a5-8926-80d527ce69ad","Type":"ContainerDied","Data":"a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d"} Nov 03 16:59:57.652373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.652168 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84" event={"ID":"9378757c-40d7-49a5-8926-80d527ce69ad","Type":"ContainerDied","Data":"ff1b77b561e267562c67dc28ed3ec4e0e69b892703a594822455deb9b0bb39eb"} Nov 03 16:59:57.652373 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.652186 4539 scope.go:122] "RemoveContainer" containerID="a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d" Nov 03 16:59:57.662780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.662756 4539 scope.go:122] "RemoveContainer" containerID="a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d" Nov 03 16:59:57.663133 ip-10-0-139-190 kubenswrapper[4539]: E1103 16:59:57.663109 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d\": container with ID starting with a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d not found: ID does not exist" containerID="a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d" Nov 03 16:59:57.663222 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.663140 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d"} err="failed to get container status \"a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d\": rpc error: code = NotFound desc = could not find container \"a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d\": container with ID starting with a41f9e8f893b7c232dc4a24fb257478671dd4210491f640678f824a08acfb11d not found: ID does not exist" Nov 03 16:59:57.730491 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.730459 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84"] Nov 03 16:59:57.750857 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:57.750828 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd067/auth-test-model-3dd067-kserve-584557d7df-7dj84"] Nov 03 16:59:58.470616 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.470522 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" path="/var/lib/kubelet/pods/58cd0726-b63e-432f-818a-0e089054a8e3/volumes" Nov 03 16:59:58.471263 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.471242 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" path="/var/lib/kubelet/pods/6d57cdfd-557b-4f81-b661-7b87125e8529/volumes" Nov 03 16:59:58.471752 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.471732 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9378757c-40d7-49a5-8926-80d527ce69ad" path="/var/lib/kubelet/pods/9378757c-40d7-49a5-8926-80d527ce69ad/volumes" Nov 03 16:59:58.472662 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.472639 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" path="/var/lib/kubelet/pods/9394bf9b-9771-4f6e-97cc-e63802c79795/volumes" Nov 03 16:59:58.811091 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.811056 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 16:59:58.926387 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.926346 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11348-2a2025"] Nov 03 16:59:58.927011 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.926991 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9378757c-40d7-49a5-8926-80d527ce69ad" containerName="main" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927013 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9378757c-40d7-49a5-8926-80d527ce69ad" containerName="main" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927052 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ca64c99b-95e9-4765-89b5-b021b710f4c3" containerName="test-curl-11342-752024" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927060 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca64c99b-95e9-4765-89b5-b021b710f4c3" containerName="test-curl-11342-752024" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927085 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927090 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927104 4539 cpu_manager.go:414] "containerMap: removing container" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" Nov 03 16:59:58.927119 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927109 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927128 4539 cpu_manager.go:414] "containerMap: removing container" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927133 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927247 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9394bf9b-9771-4f6e-97cc-e63802c79795" containerName="istio-proxy" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927258 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="58cd0726-b63e-432f-818a-0e089054a8e3" containerName="istio-proxy" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927267 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d57cdfd-557b-4f81-b661-7b87125e8529" containerName="istio-proxy" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927275 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9378757c-40d7-49a5-8926-80d527ce69ad" containerName="main" Nov 03 16:59:58.927507 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.927282 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca64c99b-95e9-4765-89b5-b021b710f4c3" containerName="test-curl-11342-752024" Nov 03 16:59:58.932357 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.932331 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-2a2025" Nov 03 16:59:58.945780 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.945756 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 16:59:58.952980 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:58.952347 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-2a2025"] Nov 03 16:59:59.019827 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.019784 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmtn\" (UniqueName: \"kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn\") pod \"test-curl-11348-2a2025\" (UID: \"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc\") " pod="opendatahub/test-curl-11348-2a2025" Nov 03 16:59:59.121796 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.121699 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmtn\" (UniqueName: \"kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn\") pod \"test-curl-11348-2a2025\" (UID: \"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc\") " pod="opendatahub/test-curl-11348-2a2025" Nov 03 16:59:59.144285 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.144245 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmtn\" (UniqueName: \"kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn\") pod \"test-curl-11348-2a2025\" (UID: \"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc\") " pod="opendatahub/test-curl-11348-2a2025" Nov 03 16:59:59.246773 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.246734 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-2a2025" Nov 03 16:59:59.557665 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.557636 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-2a2025"] Nov 03 16:59:59.558469 ip-10-0-139-190 kubenswrapper[4539]: W1103 16:59:59.558411 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0171b2c6_c8b3_4a80_84cd_3094dbaf23fc.slice/crio-8539c6c32477f6165bc196dc60a1da9fb34da394882240a75bf75ae0f915f8a6 WatchSource:0}: Error finding container 8539c6c32477f6165bc196dc60a1da9fb34da394882240a75bf75ae0f915f8a6: Status 404 returned error can't find the container with id 8539c6c32477f6165bc196dc60a1da9fb34da394882240a75bf75ae0f915f8a6 Nov 03 16:59:59.671223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.671187 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-2a2025" event={"ID":"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc","Type":"ContainerStarted","Data":"cddf9e9786db1cde97280918e1a7011cc6937eaae6bfb3c3de3145daa5176799"} Nov 03 16:59:59.671223 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.671227 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-2a2025" event={"ID":"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc","Type":"ContainerStarted","Data":"8539c6c32477f6165bc196dc60a1da9fb34da394882240a75bf75ae0f915f8a6"} Nov 03 16:59:59.722645 ip-10-0-139-190 kubenswrapper[4539]: I1103 16:59:59.722583 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11348-2a2025" podStartSLOduration=1.722565135 podStartE2EDuration="1.722565135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:59.720540822 +0000 UTC m=+2743.413811844" watchObservedRunningTime="2026-11-03 16:59:59.722565135 +0000 UTC m=+2743.415836159" Nov 03 17:00:00.311240 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.311197 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj"] Nov 03 17:00:00.325651 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.324299 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.338940 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.338897 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm"] Nov 03 17:00:00.355460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.351746 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj"] Nov 03 17:00:00.355460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.351870 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.379470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.370833 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx"] Nov 03 17:00:00.379470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.376700 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm"] Nov 03 17:00:00.379470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.376840 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.384267 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.384239 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:00:00.442920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.442861 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkt4j\" (UniqueName: \"kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.443090 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.442922 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.443090 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.442989 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjk7\" (UniqueName: \"kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7\") pod \"maas-api-key-cleanup-29895420-btkvm\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.443214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.443110 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895420-btkvm\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.443214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.443203 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.443314 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.443235 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4b2c\" (UniqueName: \"kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.443503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.443484 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6"] Nov 03 17:00:00.463483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.460860 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx"] Nov 03 17:00:00.463731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.463710 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.475947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.475915 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6"] Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545149 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdp6\" (UniqueName: \"kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545214 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkt4j\" (UniqueName: \"kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545249 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545307 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjk7\" (UniqueName: \"kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7\") pod \"maas-api-key-cleanup-29895420-btkvm\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545372 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545444 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895420-btkvm\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545520 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.545787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.545546 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4b2c\" (UniqueName: \"kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.548064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.547779 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.571038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.562252 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.571038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.563186 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895420-btkvm\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.571038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.570596 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4b2c\" (UniqueName: \"kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.574638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.574604 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkt4j\" (UniqueName: \"kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.583269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.583244 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjk7\" (UniqueName: \"kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7\") pod \"maas-api-key-cleanup-29895420-btkvm\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.632727 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.632689 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 17:00:00.635721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.635696 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 17:00:00.639216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.639192 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 17:00:00.647415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.647319 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdp6\" (UniqueName: \"kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.647582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.647540 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.648296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.648274 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.662809 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.662775 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:00.673077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.673052 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdp6\" (UniqueName: \"kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.679284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.679257 4539 generic.go:356] "Generic (PLEG): container finished" podID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerID="a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b" exitCode=0 Nov 03 17:00:00.679377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.679313 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" Nov 03 17:00:00.679377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.679335 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" event={"ID":"049c1081-44d2-440e-91bd-ec008d36c2bb","Type":"ContainerDied","Data":"a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b"} Nov 03 17:00:00.679483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.679393 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g" event={"ID":"049c1081-44d2-440e-91bd-ec008d36c2bb","Type":"ContainerDied","Data":"d0c2cceeeed10e878e8b5568a7ff3e57d0572c3df0b898f3f6274418b351a58f"} Nov 03 17:00:00.679483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.679437 4539 scope.go:122] "RemoveContainer" containerID="a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b" Nov 03 17:00:00.681138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.681110 4539 generic.go:356] "Generic (PLEG): container finished" podID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerID="ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a" exitCode=0 Nov 03 17:00:00.681446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.681155 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" event={"ID":"23f605cc-337c-4145-b4f4-a18dd70f06bf","Type":"ContainerDied","Data":"ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a"} Nov 03 17:00:00.681446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.681195 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" Nov 03 17:00:00.681446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.681227 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6" event={"ID":"23f605cc-337c-4145-b4f4-a18dd70f06bf","Type":"ContainerDied","Data":"af95c639fb87520ed55311bda7d7a3aa3677ff36b910426f73dd5845876f7e2b"} Nov 03 17:00:00.688872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.688840 4539 generic.go:356] "Generic (PLEG): container finished" podID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerID="21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3" exitCode=0 Nov 03 17:00:00.689415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.689153 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" Nov 03 17:00:00.689415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.689341 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" event={"ID":"be7a5237-fbbc-4570-85cf-a47420d9f0ec","Type":"ContainerDied","Data":"21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3"} Nov 03 17:00:00.689415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.689383 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6" event={"ID":"be7a5237-fbbc-4570-85cf-a47420d9f0ec","Type":"ContainerDied","Data":"42968c315e722b0c797a76564613d46b1ea3a532f1e5f86bcd8bbef98c54d0bc"} Nov 03 17:00:00.698125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.695390 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:00.710883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.699836 4539 scope.go:122] "RemoveContainer" containerID="a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b" Nov 03 17:00:00.710883 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:00.700200 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b\": container with ID starting with a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b not found: ID does not exist" containerID="a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b" Nov 03 17:00:00.710883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.700236 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b"} err="failed to get container status \"a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b\": rpc error: code = NotFound desc = could not find container \"a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b\": container with ID starting with a85cfe42153091fe58369a9ce4e887cf9dd98599914253b2125a4166da3a353b not found: ID does not exist" Nov 03 17:00:00.710883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.700254 4539 scope.go:122] "RemoveContainer" containerID="ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a" Nov 03 17:00:00.728821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.728601 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:00.729163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.729141 4539 scope.go:122] "RemoveContainer" containerID="ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a" Nov 03 17:00:00.729557 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:00.729536 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a\": container with ID starting with ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a not found: ID does not exist" containerID="ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a" Nov 03 17:00:00.729715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.729564 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a"} err="failed to get container status \"ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a\": rpc error: code = NotFound desc = could not find container \"ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a\": container with ID starting with ac80c39ddd5e644a05fb0cfa5ac3ea47f23de3c23f63b7e8fdeb748411b79e5a not found: ID does not exist" Nov 03 17:00:00.729715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.729582 4539 scope.go:122] "RemoveContainer" containerID="21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3" Nov 03 17:00:00.748817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.748287 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.748817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.748340 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.748817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.748372 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.748817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.748443 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752336 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752498 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752537 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752569 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752600 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752685 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752715 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752808 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752855 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752885 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752915 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.753206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.752951 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.754591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.754306 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.754591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.754348 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.754591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.754417 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.754591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.754487 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.754591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.754536 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.754906 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.754877 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "kube-api-access-hjfv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755057 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755117 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755164 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt\") pod \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\" (UID: \"be7a5237-fbbc-4570-85cf-a47420d9f0ec\") " Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755194 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755226 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo\") pod \"049c1081-44d2-440e-91bd-ec008d36c2bb\" (UID: \"049c1081-44d2-440e-91bd-ec008d36c2bb\") " Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755263 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:00.755543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.755274 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy\") pod \"23f605cc-337c-4145-b4f4-a18dd70f06bf\" (UID: \"23f605cc-337c-4145-b4f4-a18dd70f06bf\") " Nov 03 17:00:00.756491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.756289 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.756844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.756492 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.756930 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.756882 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:00.780497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.777875 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.780497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.777942 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.787962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.787907 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.789071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.789043 4539 scope.go:122] "RemoveContainer" containerID="21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3" Nov 03 17:00:00.789586 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.789548 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.789826 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.789803 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.790253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790230 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.790565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790511 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.790696 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790605 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790791 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790892 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790917 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjfv6\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-kube-api-access-hjfv6\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790941 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790958 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790974 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.790991 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f605cc-337c-4145-b4f4-a18dd70f06bf-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.791012 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.791028 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.791053 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.791086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.791069 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.792441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.791269 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:00.792441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.791387 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "kube-api-access-mvdxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:00.793699 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:00.792944 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3\": container with ID starting with 21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3 not found: ID does not exist" containerID="21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3" Nov 03 17:00:00.793699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.792985 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3"} err="failed to get container status \"21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3\": rpc error: code = NotFound desc = could not find container \"21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3\": container with ID starting with 21860e75ee36e0282e097ae4a442798ba660728cbfcd140234c1a6b459b197b3 not found: ID does not exist" Nov 03 17:00:00.793699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.793296 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.793699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.793511 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:00.793948 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.793888 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:00.808479 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.801964 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.808479 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.802903 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:00.808887 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.808824 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.814245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.814199 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:00.814979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.814939 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy" pod "23f605cc-337c-4145-b4f4-a18dd70f06bf" (UID: "23f605cc-337c-4145-b4f4-a18dd70f06bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:00.816518 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.816255 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:00.818702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.818620 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "kube-api-access-4f2kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:00.818702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.818676 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token" pod "049c1081-44d2-440e-91bd-ec008d36c2bb" (UID: "049c1081-44d2-440e-91bd-ec008d36c2bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:00.819442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.819400 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo" pod "be7a5237-fbbc-4570-85cf-a47420d9f0ec" (UID: "be7a5237-fbbc-4570-85cf-a47420d9f0ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:00.821193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.821168 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11348-2a2025"] Nov 03 17:00:00.829610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.829462 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:00.838300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.838233 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11348-2a2025"] Nov 03 17:00:00.893361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893306 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.893361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893338 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.893361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893352 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.893361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893366 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f2kt\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-kube-api-access-4f2kt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893378 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893390 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893403 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893414 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893460 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvdxm\" (UniqueName: \"kubernetes.io/projected/049c1081-44d2-440e-91bd-ec008d36c2bb-kube-api-access-mvdxm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893474 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f605cc-337c-4145-b4f4-a18dd70f06bf-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893485 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893497 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c1081-44d2-440e-91bd-ec008d36c2bb-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893509 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f605cc-337c-4145-b4f4-a18dd70f06bf-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893524 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893536 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7a5237-fbbc-4570-85cf-a47420d9f0ec-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893550 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7a5237-fbbc-4570-85cf-a47420d9f0ec-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:00.894188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:00.893562 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c1081-44d2-440e-91bd-ec008d36c2bb-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:01.021087 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.021038 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj"] Nov 03 17:00:01.104494 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:01.104455 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b63791_b0dc_46d7_a403_a20aee59a2a6.slice/crio-14445b6d403ad8cf1bad7112227006bbfb697b5605121205bb789b19140d0dcf WatchSource:0}: Error finding container 14445b6d403ad8cf1bad7112227006bbfb697b5605121205bb789b19140d0dcf: Status 404 returned error can't find the container with id 14445b6d403ad8cf1bad7112227006bbfb697b5605121205bb789b19140d0dcf Nov 03 17:00:01.105368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.105342 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm"] Nov 03 17:00:01.111097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.111071 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g"] Nov 03 17:00:01.138961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.138932 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-vgb4g"] Nov 03 17:00:01.184268 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.184245 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx"] Nov 03 17:00:01.187600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.187574 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6"] Nov 03 17:00:01.196587 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.196560 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6"] Nov 03 17:00:01.204371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.204295 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mhzq6"] Nov 03 17:00:01.209673 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:01.209639 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01972718_19ef_400b_a333_72872eed714b.slice/crio-3d4dbbef98400d1213aadd4d7f87703a968b46a461e01065dfa8600b1b7f307b WatchSource:0}: Error finding container 3d4dbbef98400d1213aadd4d7f87703a968b46a461e01065dfa8600b1b7f307b: Status 404 returned error can't find the container with id 3d4dbbef98400d1213aadd4d7f87703a968b46a461e01065dfa8600b1b7f307b Nov 03 17:00:01.211088 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:01.210984 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc951afee_0987_4757_8fe1_bc0074b3be33.slice/crio-1b2f029444c95a1623ecd73f97e0fa96dc677cf393df35fc64a94fd01af69440 WatchSource:0}: Error finding container 1b2f029444c95a1623ecd73f97e0fa96dc677cf393df35fc64a94fd01af69440: Status 404 returned error can't find the container with id 1b2f029444c95a1623ecd73f97e0fa96dc677cf393df35fc64a94fd01af69440 Nov 03 17:00:01.247511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.247440 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6"] Nov 03 17:00:01.252645 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.252620 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7spf6"] Nov 03 17:00:01.697503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.697469 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerStarted","Data":"55554933bcd65b2e6b2b3c988f3b8c96937ed8fe7ca332982eb17e7623e6f17b"} Nov 03 17:00:01.698051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.697524 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerStarted","Data":"9602d3872bd84344a8cd5c01d61356ea258a5ab312b3220ad14ec6f739c0a74e"} Nov 03 17:00:01.701210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.700471 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerStarted","Data":"54d7b774fbbe21ff437d6e0778117027099c803a379c3b767f9b04ac2f53fd49"} Nov 03 17:00:01.701210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.700520 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerStarted","Data":"3d4dbbef98400d1213aadd4d7f87703a968b46a461e01065dfa8600b1b7f307b"} Nov 03 17:00:01.702899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.702874 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerStarted","Data":"1b2f029444c95a1623ecd73f97e0fa96dc677cf393df35fc64a94fd01af69440"} Nov 03 17:00:01.704401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.704379 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerStarted","Data":"14445b6d403ad8cf1bad7112227006bbfb697b5605121205bb789b19140d0dcf"} Nov 03 17:00:01.705593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.705566 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11348-2a2025" podUID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" containerName="test-curl-11348-2a2025" containerID="cri-o://cddf9e9786db1cde97280918e1a7011cc6937eaae6bfb3c3de3145daa5176799" gracePeriod=2 Nov 03 17:00:01.759750 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:01.759661 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-2a2025\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" pod="opendatahub/test-curl-11348-2a2025" Nov 03 17:00:01.811654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.811594 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" podStartSLOduration=1.811574579 podStartE2EDuration="1.811574579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:01.747978139 +0000 UTC m=+2745.441249161" watchObservedRunningTime="2026-11-03 17:00:01.811574579 +0000 UTC m=+2745.504845602" Nov 03 17:00:01.812777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:01.812741 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" podStartSLOduration=1.812731947 podStartE2EDuration="1.812731947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:01.80732198 +0000 UTC m=+2745.500593003" watchObservedRunningTime="2026-11-03 17:00:01.812731947 +0000 UTC m=+2745.506002968" Nov 03 17:00:02.471196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.471164 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" path="/var/lib/kubelet/pods/049c1081-44d2-440e-91bd-ec008d36c2bb/volumes" Nov 03 17:00:02.471677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.471663 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" path="/var/lib/kubelet/pods/23f605cc-337c-4145-b4f4-a18dd70f06bf/volumes" Nov 03 17:00:02.472075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.472063 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" path="/var/lib/kubelet/pods/be7a5237-fbbc-4570-85cf-a47420d9f0ec/volumes" Nov 03 17:00:02.622535 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.622502 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:02.753471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.753356 4539 generic.go:356] "Generic (PLEG): container finished" podID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerID="55554933bcd65b2e6b2b3c988f3b8c96937ed8fe7ca332982eb17e7623e6f17b" exitCode=60 Nov 03 17:00:02.753947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.753570 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerDied","Data":"55554933bcd65b2e6b2b3c988f3b8c96937ed8fe7ca332982eb17e7623e6f17b"} Nov 03 17:00:02.754074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.754014 4539 scope.go:122] "RemoveContainer" containerID="55554933bcd65b2e6b2b3c988f3b8c96937ed8fe7ca332982eb17e7623e6f17b" Nov 03 17:00:02.758206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.756650 4539 generic.go:356] "Generic (PLEG): container finished" podID="01972718-19ef-400b-a333-72872eed714b" containerID="54d7b774fbbe21ff437d6e0778117027099c803a379c3b767f9b04ac2f53fd49" exitCode=60 Nov 03 17:00:02.758206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.756759 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerDied","Data":"54d7b774fbbe21ff437d6e0778117027099c803a379c3b767f9b04ac2f53fd49"} Nov 03 17:00:02.758206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.757153 4539 scope.go:122] "RemoveContainer" containerID="54d7b774fbbe21ff437d6e0778117027099c803a379c3b767f9b04ac2f53fd49" Nov 03 17:00:02.759600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.759574 4539 generic.go:356] "Generic (PLEG): container finished" podID="c951afee-0987-4757-8fe1-bc0074b3be33" containerID="e9cfeef5c5ae1bb85649a0d0b00f608e415195ec65b50df293f8018152e1d849" exitCode=60 Nov 03 17:00:02.759769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.759658 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerDied","Data":"e9cfeef5c5ae1bb85649a0d0b00f608e415195ec65b50df293f8018152e1d849"} Nov 03 17:00:02.760026 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.760008 4539 scope.go:122] "RemoveContainer" containerID="e9cfeef5c5ae1bb85649a0d0b00f608e415195ec65b50df293f8018152e1d849" Nov 03 17:00:02.762375 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.762349 4539 generic.go:356] "Generic (PLEG): container finished" podID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerID="03d09dfccfc3ac20c58dd6673f3501bd246dbcf66a24686c451e3dfe3952c707" exitCode=60 Nov 03 17:00:02.762574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.762554 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerDied","Data":"03d09dfccfc3ac20c58dd6673f3501bd246dbcf66a24686c451e3dfe3952c707"} Nov 03 17:00:02.762927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:02.762909 4539 scope.go:122] "RemoveContainer" containerID="03d09dfccfc3ac20c58dd6673f3501bd246dbcf66a24686c451e3dfe3952c707" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.074796 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb"] Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075785 4539 cpu_manager.go:414] "containerMap: removing container" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075807 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075831 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" containerName="test-curl-11348-2a2025" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075839 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" containerName="test-curl-11348-2a2025" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075863 4539 cpu_manager.go:414] "containerMap: removing container" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075871 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075889 4539 cpu_manager.go:414] "containerMap: removing container" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.075897 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.076055 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7a5237-fbbc-4570-85cf-a47420d9f0ec" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.076071 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" containerName="test-curl-11348-2a2025" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.076085 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f605cc-337c-4145-b4f4-a18dd70f06bf" containerName="istio-proxy" Nov 03 17:00:03.080450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.076096 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="049c1081-44d2-440e-91bd-ec008d36c2bb" containerName="istio-proxy" Nov 03 17:00:03.183934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.183892 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb"] Nov 03 17:00:03.184268 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.184252 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.211606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.204622 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2492c4-openshift-default-dockercfg-4dlv8\"" Nov 03 17:00:03.238600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.235678 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:03.264477 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.264442 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.264761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.264741 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.264923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.264909 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.265149 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.265135 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.265330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.265315 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.265513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.265499 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.265869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.265749 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.265869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.265841 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.266070 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.265883 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5hh\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.367968 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368026 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368058 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368135 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368220 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368259 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5hh\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368345 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368376 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368458 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.368959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.368643 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.370035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.369970 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.370201 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.370176 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.370347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.370202 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.370834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.370810 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.373229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.373176 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.374769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.374743 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.399071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.399034 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5hh\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.411889 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.411344 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token\") pod \"e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.545797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.545757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:03.565830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.565793 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:03.798836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.798474 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:03.821386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.820789 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerStarted","Data":"54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73"} Nov 03 17:00:03.823745 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.823636 4539 generic.go:356] "Generic (PLEG): container finished" podID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" containerID="cddf9e9786db1cde97280918e1a7011cc6937eaae6bfb3c3de3145daa5176799" exitCode=137 Nov 03 17:00:03.826475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.826149 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerStarted","Data":"4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae"} Nov 03 17:00:03.831450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.830945 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerStarted","Data":"73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23"} Nov 03 17:00:03.834786 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:03.834214 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerStarted","Data":"1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3"} Nov 03 17:00:04.023822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.023788 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb"] Nov 03 17:00:04.081181 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:04.081140 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80150068_7b02_4d59_84f7_1e23530dd995.slice/crio-ef848f0c19649eaab91cc463088bd58c71b88d4ddc61f03441de838f13c84381 WatchSource:0}: Error finding container ef848f0c19649eaab91cc463088bd58c71b88d4ddc61f03441de838f13c84381: Status 404 returned error can't find the container with id ef848f0c19649eaab91cc463088bd58c71b88d4ddc61f03441de838f13c84381 Nov 03 17:00:04.083946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.083913 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:04.084086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.084000 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:04.084086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.084049 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:04.100988 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.100951 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-2a2025" Nov 03 17:00:04.110942 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.110899 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn\" (UniqueName: \"kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn\") pod \"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc\" (UID: \"0171b2c6-c8b3-4a80-84cd-3094dbaf23fc\") " Nov 03 17:00:04.135005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.134963 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn" pod "0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" (UID: "0171b2c6-c8b3-4a80-84cd-3094dbaf23fc"). InnerVolumeSpecName "kube-api-access-dmmtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:04.148599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.147065 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" podStartSLOduration=4.147038058 podStartE2EDuration="4.147038058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:04.03869597 +0000 UTC m=+2747.731966992" watchObservedRunningTime="2026-11-03 17:00:04.147038058 +0000 UTC m=+2747.840309081" Nov 03 17:00:04.149804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.149753 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" podStartSLOduration=4.149735704 podStartE2EDuration="4.149735704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:04.134749954 +0000 UTC m=+2747.828020977" watchObservedRunningTime="2026-11-03 17:00:04.149735704 +0000 UTC m=+2747.843006727" Nov 03 17:00:04.216683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.216640 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmtn\" (UniqueName: \"kubernetes.io/projected/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc-kube-api-access-dmmtn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:04.474293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.474253 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0171b2c6-c8b3-4a80-84cd-3094dbaf23fc" path="/var/lib/kubelet/pods/0171b2c6-c8b3-4a80-84cd-3094dbaf23fc/volumes" Nov 03 17:00:04.673451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.672814 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" podUID="9388c726-b496-47ff-ba87-67ee3d5e3825" containerName="main" containerID="cri-o://03b8b1bd217aacc3e8135d735830915b4492406d9de3e6d7283cc7141d128178" gracePeriod=45 Nov 03 17:00:04.848605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.848474 4539 scope.go:122] "RemoveContainer" containerID="cddf9e9786db1cde97280918e1a7011cc6937eaae6bfb3c3de3145daa5176799" Nov 03 17:00:04.849626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.849513 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-2a2025" Nov 03 17:00:04.854496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.852765 4539 generic.go:356] "Generic (PLEG): container finished" podID="01972718-19ef-400b-a333-72872eed714b" containerID="4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae" exitCode=60 Nov 03 17:00:04.854496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.852853 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerDied","Data":"4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae"} Nov 03 17:00:04.854496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.853175 4539 scope.go:122] "RemoveContainer" containerID="4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae" Nov 03 17:00:04.854496 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:04.853535 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx_odh-ai-gateway-infra(01972718-19ef-400b-a333-72872eed714b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" podUID="01972718-19ef-400b-a333-72872eed714b" Nov 03 17:00:04.859000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.858688 4539 generic.go:356] "Generic (PLEG): container finished" podID="c951afee-0987-4757-8fe1-bc0074b3be33" containerID="73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23" exitCode=60 Nov 03 17:00:04.859000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.858767 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerDied","Data":"73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23"} Nov 03 17:00:04.859255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.859160 4539 scope.go:122] "RemoveContainer" containerID="73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23" Nov 03 17:00:04.859463 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:04.859439 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6_odh-ai-gateway-infra(c951afee-0987-4757-8fe1-bc0074b3be33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" Nov 03 17:00:04.864287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.864262 4539 scope.go:122] "RemoveContainer" containerID="54d7b774fbbe21ff437d6e0778117027099c803a379c3b767f9b04ac2f53fd49" Nov 03 17:00:04.864823 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.864792 4539 generic.go:356] "Generic (PLEG): container finished" podID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerID="1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3" exitCode=60 Nov 03 17:00:04.864911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.864879 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerDied","Data":"1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3"} Nov 03 17:00:04.865247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.865229 4539 scope.go:122] "RemoveContainer" containerID="1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3" Nov 03 17:00:04.866555 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:04.865933 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895420-btkvm_odh-ai-gateway-infra(a8b63791-b0dc-46d7-a403-a20aee59a2a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" Nov 03 17:00:04.867038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.867016 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" event={"ID":"80150068-7b02-4d59-84f7-1e23530dd995","Type":"ContainerStarted","Data":"9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e"} Nov 03 17:00:04.867106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.867071 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" event={"ID":"80150068-7b02-4d59-84f7-1e23530dd995","Type":"ContainerStarted","Data":"ef848f0c19649eaab91cc463088bd58c71b88d4ddc61f03441de838f13c84381"} Nov 03 17:00:04.871466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.871369 4539 generic.go:356] "Generic (PLEG): container finished" podID="9388c726-b496-47ff-ba87-67ee3d5e3825" containerID="03b8b1bd217aacc3e8135d735830915b4492406d9de3e6d7283cc7141d128178" exitCode=0 Nov 03 17:00:04.871719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.871659 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" event={"ID":"9388c726-b496-47ff-ba87-67ee3d5e3825","Type":"ContainerDied","Data":"03b8b1bd217aacc3e8135d735830915b4492406d9de3e6d7283cc7141d128178"} Nov 03 17:00:04.874104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.874080 4539 generic.go:356] "Generic (PLEG): container finished" podID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerID="54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73" exitCode=60 Nov 03 17:00:04.874190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.874160 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerDied","Data":"54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73"} Nov 03 17:00:04.874538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.874522 4539 scope.go:122] "RemoveContainer" containerID="54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73" Nov 03 17:00:04.874815 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:04.874795 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj_odh-ai-gateway-infra(bae52e79-1f96-47e8-86c6-33dfc19a5974)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" Nov 03 17:00:04.882741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.882181 4539 scope.go:122] "RemoveContainer" containerID="e9cfeef5c5ae1bb85649a0d0b00f608e415195ec65b50df293f8018152e1d849" Nov 03 17:00:04.922235 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.922214 4539 scope.go:122] "RemoveContainer" containerID="03d09dfccfc3ac20c58dd6673f3501bd246dbcf66a24686c451e3dfe3952c707" Nov 03 17:00:04.937593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.937511 4539 scope.go:122] "RemoveContainer" containerID="55554933bcd65b2e6b2b3c988f3b8c96937ed8fe7ca332982eb17e7623e6f17b" Nov 03 17:00:04.999075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:04.999047 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 17:00:05.008021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.007959 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podStartSLOduration=2.007940758 podStartE2EDuration="2.007940758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:05.001812974 +0000 UTC m=+2748.695083995" watchObservedRunningTime="2026-11-03 17:00:05.007940758 +0000 UTC m=+2748.701211783" Nov 03 17:00:05.032821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.032789 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir\") pod \"9388c726-b496-47ff-ba87-67ee3d5e3825\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " Nov 03 17:00:05.032993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.032930 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs\" (UniqueName: \"kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs\") pod \"9388c726-b496-47ff-ba87-67ee3d5e3825\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " Nov 03 17:00:05.032993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.032965 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc\" (UniqueName: \"kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc\") pod \"9388c726-b496-47ff-ba87-67ee3d5e3825\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " Nov 03 17:00:05.033088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033060 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir" pod "9388c726-b496-47ff-ba87-67ee3d5e3825" (UID: "9388c726-b496-47ff-ba87-67ee3d5e3825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:05.033130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033103 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache\") pod \"9388c726-b496-47ff-ba87-67ee3d5e3825\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " Nov 03 17:00:05.033168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033131 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home\") pod \"9388c726-b496-47ff-ba87-67ee3d5e3825\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " Nov 03 17:00:05.033168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033159 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm\") pod \"9388c726-b496-47ff-ba87-67ee3d5e3825\" (UID: \"9388c726-b496-47ff-ba87-67ee3d5e3825\") " Nov 03 17:00:05.033377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033348 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache" pod "9388c726-b496-47ff-ba87-67ee3d5e3825" (UID: "9388c726-b496-47ff-ba87-67ee3d5e3825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:05.033377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033361 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home" pod "9388c726-b496-47ff-ba87-67ee3d5e3825" (UID: "9388c726-b496-47ff-ba87-67ee3d5e3825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:05.033922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033901 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:05.033992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033931 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:05.033992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.033945 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:05.035451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.035407 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc" pod "9388c726-b496-47ff-ba87-67ee3d5e3825" (UID: "9388c726-b496-47ff-ba87-67ee3d5e3825"). InnerVolumeSpecName "kube-api-access-5dhmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:05.035548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.035457 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm" pod "9388c726-b496-47ff-ba87-67ee3d5e3825" (UID: "9388c726-b496-47ff-ba87-67ee3d5e3825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:05.035548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.035501 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs" pod "9388c726-b496-47ff-ba87-67ee3d5e3825" (UID: "9388c726-b496-47ff-ba87-67ee3d5e3825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:05.136204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.136108 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9388c726-b496-47ff-ba87-67ee3d5e3825-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:05.136204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.136153 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9388c726-b496-47ff-ba87-67ee3d5e3825-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:05.136204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.136172 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhmc\" (UniqueName: \"kubernetes.io/projected/9388c726-b496-47ff-ba87-67ee3d5e3825-kube-api-access-5dhmc\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:05.566608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.566575 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:05.572720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.572691 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:05.884551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.884466 4539 scope.go:122] "RemoveContainer" containerID="4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae" Nov 03 17:00:05.885015 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:05.884824 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx_odh-ai-gateway-infra(01972718-19ef-400b-a333-72872eed714b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" podUID="01972718-19ef-400b-a333-72872eed714b" Nov 03 17:00:05.886350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.886326 4539 scope.go:122] "RemoveContainer" containerID="73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23" Nov 03 17:00:05.886683 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:05.886662 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6_odh-ai-gateway-infra(c951afee-0987-4757-8fe1-bc0074b3be33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" Nov 03 17:00:05.888168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.888149 4539 scope.go:122] "RemoveContainer" containerID="1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3" Nov 03 17:00:05.888416 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:05.888387 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895420-btkvm_odh-ai-gateway-infra(a8b63791-b0dc-46d7-a403-a20aee59a2a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" Nov 03 17:00:05.889514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.889491 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" Nov 03 17:00:05.889621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.889539 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6" event={"ID":"9388c726-b496-47ff-ba87-67ee3d5e3825","Type":"ContainerDied","Data":"d4acac9a345c239a3768f850384e20f0b75bec3cf0c2ac60c25bf3194cc82e43"} Nov 03 17:00:05.889621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.889586 4539 scope.go:122] "RemoveContainer" containerID="03b8b1bd217aacc3e8135d735830915b4492406d9de3e6d7283cc7141d128178" Nov 03 17:00:05.891919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.891693 4539 scope.go:122] "RemoveContainer" containerID="54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73" Nov 03 17:00:05.892023 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:05.891955 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj_odh-ai-gateway-infra(bae52e79-1f96-47e8-86c6-33dfc19a5974)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" Nov 03 17:00:05.893322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.893304 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:05.897184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.897167 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:00:05.986540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.986499 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6"] Nov 03 17:00:05.992182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:05.992144 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2b49df/auth-test-model-2b49df-kserve-76b56758d9-nsrl6"] Nov 03 17:00:06.176273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.176172 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z"] Nov 03 17:00:06.176494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.176468 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" podUID="1987c4e0-a459-4fdf-9489-c7943b68314e" containerName="maas-api" containerID="cri-o://1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e" gracePeriod=30 Nov 03 17:00:06.315270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.315238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:06.569095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.569071 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9388c726-b496-47ff-ba87-67ee3d5e3825" path="/var/lib/kubelet/pods/9388c726-b496-47ff-ba87-67ee3d5e3825/volumes" Nov 03 17:00:06.592397 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.592365 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 17:00:06.704293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.703140 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf\" (UniqueName: \"kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf\") pod \"1987c4e0-a459-4fdf-9489-c7943b68314e\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " Nov 03 17:00:06.704293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.703326 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs\") pod \"1987c4e0-a459-4fdf-9489-c7943b68314e\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " Nov 03 17:00:06.704293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.703405 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls\") pod \"1987c4e0-a459-4fdf-9489-c7943b68314e\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " Nov 03 17:00:06.704293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.703562 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca\") pod \"1987c4e0-a459-4fdf-9489-c7943b68314e\" (UID: \"1987c4e0-a459-4fdf-9489-c7943b68314e\") " Nov 03 17:00:06.705236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.704916 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca" pod "1987c4e0-a459-4fdf-9489-c7943b68314e" (UID: "1987c4e0-a459-4fdf-9489-c7943b68314e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:06.735492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.718904 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls" pod "1987c4e0-a459-4fdf-9489-c7943b68314e" (UID: "1987c4e0-a459-4fdf-9489-c7943b68314e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:06.735492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.722405 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf" pod "1987c4e0-a459-4fdf-9489-c7943b68314e" (UID: "1987c4e0-a459-4fdf-9489-c7943b68314e"). InnerVolumeSpecName "kube-api-access-vhwkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:06.751454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.745066 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs" pod "1987c4e0-a459-4fdf-9489-c7943b68314e" (UID: "1987c4e0-a459-4fdf-9489-c7943b68314e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:06.805501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.805106 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:06.805501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.805137 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1987c4e0-a459-4fdf-9489-c7943b68314e-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:06.805501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.805153 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1987c4e0-a459-4fdf-9489-c7943b68314e-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:06.805501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.805166 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwkf\" (UniqueName: \"kubernetes.io/projected/1987c4e0-a459-4fdf-9489-c7943b68314e-kube-api-access-vhwkf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:06.901752 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.901714 4539 generic.go:356] "Generic (PLEG): container finished" podID="1987c4e0-a459-4fdf-9489-c7943b68314e" containerID="1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e" exitCode=0 Nov 03 17:00:06.902223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.901780 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" Nov 03 17:00:06.902223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.901806 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" event={"ID":"1987c4e0-a459-4fdf-9489-c7943b68314e","Type":"ContainerDied","Data":"1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e"} Nov 03 17:00:06.902223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.901869 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z" event={"ID":"1987c4e0-a459-4fdf-9489-c7943b68314e","Type":"ContainerDied","Data":"d6540993ff85cd2570b80087d975b898c1e3f029e33765c5111a0b231c47da6e"} Nov 03 17:00:06.902223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.901898 4539 scope.go:122] "RemoveContainer" containerID="1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e" Nov 03 17:00:06.916044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.916021 4539 scope.go:122] "RemoveContainer" containerID="1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e" Nov 03 17:00:06.916390 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:06.916372 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e\": container with ID starting with 1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e not found: ID does not exist" containerID="1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e" Nov 03 17:00:06.916496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.916400 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e"} err="failed to get container status \"1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e\": rpc error: code = NotFound desc = could not find container \"1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e\": container with ID starting with 1bdaedf0cf379a36620015e0e26972e88788f6f868e0078092041e2eea1ba11e not found: ID does not exist" Nov 03 17:00:06.955453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.955405 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z"] Nov 03 17:00:06.979564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:06.979537 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c1837e-749779ccc7-4zq5z"] Nov 03 17:00:07.236211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.236101 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:07.423655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.423620 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11345-31a005"] Nov 03 17:00:07.424199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.424187 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9388c726-b496-47ff-ba87-67ee3d5e3825" containerName="main" Nov 03 17:00:07.424245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.424201 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9388c726-b496-47ff-ba87-67ee3d5e3825" containerName="main" Nov 03 17:00:07.424245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.424217 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1987c4e0-a459-4fdf-9489-c7943b68314e" containerName="maas-api" Nov 03 17:00:07.424245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.424223 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1987c4e0-a459-4fdf-9489-c7943b68314e" containerName="maas-api" Nov 03 17:00:07.424349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.424336 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9388c726-b496-47ff-ba87-67ee3d5e3825" containerName="main" Nov 03 17:00:07.424349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.424345 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1987c4e0-a459-4fdf-9489-c7943b68314e" containerName="maas-api" Nov 03 17:00:07.428953 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.428936 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:07.447167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.447144 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 17:00:07.449917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.449890 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11345-31a005"] Nov 03 17:00:07.513191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.513104 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhcj\" (UniqueName: \"kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj\") pod \"test-curl-11345-31a005\" (UID: \"c92b32cd-efeb-4455-bff8-8b38e22e6881\") " pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:07.614269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.614234 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhcj\" (UniqueName: \"kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj\") pod \"test-curl-11345-31a005\" (UID: \"c92b32cd-efeb-4455-bff8-8b38e22e6881\") " pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:07.681193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.681156 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhcj\" (UniqueName: \"kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj\") pod \"test-curl-11345-31a005\" (UID: \"c92b32cd-efeb-4455-bff8-8b38e22e6881\") " pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:07.743927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.743899 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:07.807593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:07.807514 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:08.073250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.073208 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11345-31a005"] Nov 03 17:00:08.073622 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:08.073363 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc92b32cd_efeb_4455_bff8_8b38e22e6881.slice/crio-bc1f28b925af7de8aafe32225e70d5ea3b04f8641d41e935ba8827f2805e2ee5 WatchSource:0}: Error finding container bc1f28b925af7de8aafe32225e70d5ea3b04f8641d41e935ba8827f2805e2ee5: Status 404 returned error can't find the container with id bc1f28b925af7de8aafe32225e70d5ea3b04f8641d41e935ba8827f2805e2ee5 Nov 03 17:00:08.080684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.080648 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr"] Nov 03 17:00:08.083540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.083509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:08.090104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.090088 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.103738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.102145 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr"] Nov 03 17:00:08.117681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.117639 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:00:08.124371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.124341 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.124525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.124389 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.124525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.124406 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28mt\" (UniqueName: \"kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.226195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.226161 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.226446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.226203 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.226446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.226222 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28mt\" (UniqueName: \"kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.226997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.226972 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.229095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.229066 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.248458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.248406 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28mt\" (UniqueName: \"kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt\") pod \"maas-api-revoke-sub-411f3e66-7w5hr\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.413391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.413291 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:08.471048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.471011 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1987c4e0-a459-4fdf-9489-c7943b68314e" path="/var/lib/kubelet/pods/1987c4e0-a459-4fdf-9489-c7943b68314e/volumes" Nov 03 17:00:08.738846 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:08.738810 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27168524_3ef5_4c01_aa8b_958ba9b4e0c7.slice/crio-b063d300f78f286eb9a0b9da77322759d5c93fff7a4d8ef1b3e39688857fcb99 WatchSource:0}: Error finding container b063d300f78f286eb9a0b9da77322759d5c93fff7a4d8ef1b3e39688857fcb99: Status 404 returned error can't find the container with id b063d300f78f286eb9a0b9da77322759d5c93fff7a4d8ef1b3e39688857fcb99 Nov 03 17:00:08.739536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.739510 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr"] Nov 03 17:00:08.925638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.925607 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11345-31a005" event={"ID":"c92b32cd-efeb-4455-bff8-8b38e22e6881","Type":"ContainerStarted","Data":"83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010"} Nov 03 17:00:08.925777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.925648 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11345-31a005" event={"ID":"c92b32cd-efeb-4455-bff8-8b38e22e6881","Type":"ContainerStarted","Data":"bc1f28b925af7de8aafe32225e70d5ea3b04f8641d41e935ba8827f2805e2ee5"} Nov 03 17:00:08.927590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.927563 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" event={"ID":"27168524-3ef5-4c01-aa8b-958ba9b4e0c7","Type":"ContainerStarted","Data":"b063d300f78f286eb9a0b9da77322759d5c93fff7a4d8ef1b3e39688857fcb99"} Nov 03 17:00:08.977577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:08.977516 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11345-31a005" podStartSLOduration=1.977498909 podStartE2EDuration="1.977498909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:08.975474483 +0000 UTC m=+2752.668745507" watchObservedRunningTime="2026-11-03 17:00:08.977498909 +0000 UTC m=+2752.670769963" Nov 03 17:00:09.073407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.073378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:09.531581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.531552 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:09.651657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.651622 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc"] Nov 03 17:00:09.655769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.655745 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.685701 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.685668 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc"] Nov 03 17:00:09.701466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.701417 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2492c4\"" Nov 03 17:00:09.701646 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.701624 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2492c4-dockercfg-mpvl4\"" Nov 03 17:00:09.745067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.745015 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpmx\" (UniqueName: \"kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx\") pod \"payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.745067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.745063 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.841500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.828351 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq"] Nov 03 17:00:09.862491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.846253 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpmx\" (UniqueName: \"kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx\") pod \"payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.862491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.846299 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.862491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.847501 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.893499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.883812 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:09.893499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.887211 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq"] Nov 03 17:00:09.951110 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.937171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:09.951110 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.947277 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbj4n\" (UniqueName: \"kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n\") pod \"payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:09.951110 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.947379 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:09.953238 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.953143 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpmx\" (UniqueName: \"kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx\") pod \"payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:09.954170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.954141 4539 generic.go:356] "Generic (PLEG): container finished" podID="27168524-3ef5-4c01-aa8b-958ba9b4e0c7" containerID="0ae333b52e15424a3bf667f5c7d84fbc886e23713a318fefe2aa4a6a2f5aa725" exitCode=0 Nov 03 17:00:09.954984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.954960 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" event={"ID":"27168524-3ef5-4c01-aa8b-958ba9b4e0c7","Type":"ContainerDied","Data":"0ae333b52e15424a3bf667f5c7d84fbc886e23713a318fefe2aa4a6a2f5aa725"} Nov 03 17:00:09.969973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:09.969951 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:10.048454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.048399 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbj4n\" (UniqueName: \"kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n\") pod \"payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:10.048592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.048573 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:10.052567 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.052533 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:10.075111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.074184 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbj4n\" (UniqueName: \"kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n\") pod \"payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:10.190076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.190005 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:10.242809 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.242765 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11345-31a005"] Nov 03 17:00:10.251517 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.251479 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11345-31a005"] Nov 03 17:00:10.276210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.276165 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:10.316037 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.316005 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc"] Nov 03 17:00:10.318713 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:10.317763 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f0db2d_138f_467c_9981_42b8a42ed55c.slice/crio-da43bb4266150c5a1ab8d27101a46fb2a17e705612d58ddbb0e14af4284af6e9 WatchSource:0}: Error finding container da43bb4266150c5a1ab8d27101a46fb2a17e705612d58ddbb0e14af4284af6e9: Status 404 returned error can't find the container with id da43bb4266150c5a1ab8d27101a46fb2a17e705612d58ddbb0e14af4284af6e9 Nov 03 17:00:10.629944 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.629906 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq"] Nov 03 17:00:10.634018 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:10.633983 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589dfe3f_d37b_4c6e_8cfb_ed68c49c0e5d.slice/crio-02ba263a7171faf49d67e02409b25e02edfbf0ececc407e6ad00545a348c43c0 WatchSource:0}: Error finding container 02ba263a7171faf49d67e02409b25e02edfbf0ececc407e6ad00545a348c43c0: Status 404 returned error can't find the container with id 02ba263a7171faf49d67e02409b25e02edfbf0ececc407e6ad00545a348c43c0 Nov 03 17:00:10.963009 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.962914 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" event={"ID":"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d","Type":"ContainerStarted","Data":"abbd64f7b09dfa7cf8286782fd77db38ee54a2c7c41f4691d8afd3589deed2f4"} Nov 03 17:00:10.963009 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.962966 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" event={"ID":"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d","Type":"ContainerStarted","Data":"02ba263a7171faf49d67e02409b25e02edfbf0ececc407e6ad00545a348c43c0"} Nov 03 17:00:10.963233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.963009 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:10.964770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.964739 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" event={"ID":"c8f0db2d-138f-467c-9981-42b8a42ed55c","Type":"ContainerStarted","Data":"efa876c4866f64006666c3ec90927ba8f97a150a56be41e5269307637ce2bd48"} Nov 03 17:00:10.964882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.964783 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" event={"ID":"c8f0db2d-138f-467c-9981-42b8a42ed55c","Type":"ContainerStarted","Data":"da43bb4266150c5a1ab8d27101a46fb2a17e705612d58ddbb0e14af4284af6e9"} Nov 03 17:00:10.964882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.964845 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11345-31a005" podUID="c92b32cd-efeb-4455-bff8-8b38e22e6881" containerName="test-curl-11345-31a005" containerID="cri-o://83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010" gracePeriod=2 Nov 03 17:00:10.965012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:10.964953 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:11.016271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.016204 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" podStartSLOduration=2.016185327 podStartE2EDuration="2.016185327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:11.01346078 +0000 UTC m=+2754.706731799" watchObservedRunningTime="2026-11-03 17:00:11.016185327 +0000 UTC m=+2754.709456350" Nov 03 17:00:11.059034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.058972 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" podStartSLOduration=2.058950066 podStartE2EDuration="2.058950066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:11.054644002 +0000 UTC m=+2754.747915025" watchObservedRunningTime="2026-11-03 17:00:11.058950066 +0000 UTC m=+2754.752221089" Nov 03 17:00:11.144029 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.143998 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:11.163833 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.163806 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt\" (UniqueName: \"kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt\") pod \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " Nov 03 17:00:11.163972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.163895 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca\") pod \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " Nov 03 17:00:11.163972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.163951 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca\") pod \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\" (UID: \"27168524-3ef5-4c01-aa8b-958ba9b4e0c7\") " Nov 03 17:00:11.164707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.164351 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca" pod "27168524-3ef5-4c01-aa8b-958ba9b4e0c7" (UID: "27168524-3ef5-4c01-aa8b-958ba9b4e0c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:11.166621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.166595 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca" pod "27168524-3ef5-4c01-aa8b-958ba9b4e0c7" (UID: "27168524-3ef5-4c01-aa8b-958ba9b4e0c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:11.167064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.167031 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt" pod "27168524-3ef5-4c01-aa8b-958ba9b4e0c7" (UID: "27168524-3ef5-4c01-aa8b-958ba9b4e0c7"). InnerVolumeSpecName "kube-api-access-q28mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:11.265325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.265237 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28mt\" (UniqueName: \"kubernetes.io/projected/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-kube-api-access-q28mt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:11.265325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.265271 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:11.265325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.265281 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27168524-3ef5-4c01-aa8b-958ba9b4e0c7-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:11.533837 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.533808 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:11.729233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.729193 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:11.973467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.973362 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" Nov 03 17:00:11.973467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.973406 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr" event={"ID":"27168524-3ef5-4c01-aa8b-958ba9b4e0c7","Type":"ContainerDied","Data":"b063d300f78f286eb9a0b9da77322759d5c93fff7a4d8ef1b3e39688857fcb99"} Nov 03 17:00:11.973653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:11.973477 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b063d300f78f286eb9a0b9da77322759d5c93fff7a4d8ef1b3e39688857fcb99" Nov 03 17:00:12.354676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:12.354641 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:12.556007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:12.555978 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:12.911158 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:12.911063 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:13.199052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199019 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11339-de8381"] Nov 03 17:00:13.199692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199672 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c92b32cd-efeb-4455-bff8-8b38e22e6881" containerName="test-curl-11345-31a005" Nov 03 17:00:13.199788 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199695 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92b32cd-efeb-4455-bff8-8b38e22e6881" containerName="test-curl-11345-31a005" Nov 03 17:00:13.199788 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199724 4539 cpu_manager.go:414] "containerMap: removing container" podUID="27168524-3ef5-4c01-aa8b-958ba9b4e0c7" containerName="revoke-keys" Nov 03 17:00:13.199788 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199733 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27168524-3ef5-4c01-aa8b-958ba9b4e0c7" containerName="revoke-keys" Nov 03 17:00:13.199928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199889 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="27168524-3ef5-4c01-aa8b-958ba9b4e0c7" containerName="revoke-keys" Nov 03 17:00:13.199928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.199910 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92b32cd-efeb-4455-bff8-8b38e22e6881" containerName="test-curl-11345-31a005" Nov 03 17:00:13.213782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.213756 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:13.217687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.217665 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:13.230783 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.230755 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11339-de8381"] Nov 03 17:00:13.267733 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.267700 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:13.294458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.294416 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj\" (UniqueName: \"kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj\") pod \"c92b32cd-efeb-4455-bff8-8b38e22e6881\" (UID: \"c92b32cd-efeb-4455-bff8-8b38e22e6881\") " Nov 03 17:00:13.294793 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.294772 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2\") pod \"test-curl-11339-de8381\" (UID: \"0ab08aef-2d26-41d6-b780-6c158485d372\") " pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:13.296624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.296602 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj" pod "c92b32cd-efeb-4455-bff8-8b38e22e6881" (UID: "c92b32cd-efeb-4455-bff8-8b38e22e6881"). InnerVolumeSpecName "kube-api-access-frhcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:13.395830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.395798 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2\") pod \"test-curl-11339-de8381\" (UID: \"0ab08aef-2d26-41d6-b780-6c158485d372\") " pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:13.396273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.395962 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhcj\" (UniqueName: \"kubernetes.io/projected/c92b32cd-efeb-4455-bff8-8b38e22e6881-kube-api-access-frhcj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:13.418966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.418924 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2\") pod \"test-curl-11339-de8381\" (UID: \"0ab08aef-2d26-41d6-b780-6c158485d372\") " pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:13.537785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.537743 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:13.842772 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:13.842663 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11339-de8381"] Nov 03 17:00:13.845568 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:13.845537 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ab08aef_2d26_41d6_b780_6c158485d372.slice/crio-9c24c7f7256b366dae642584b40c8125a5a67d172ff01b5a33f767dc9c2a230d WatchSource:0}: Error finding container 9c24c7f7256b366dae642584b40c8125a5a67d172ff01b5a33f767dc9c2a230d: Status 404 returned error can't find the container with id 9c24c7f7256b366dae642584b40c8125a5a67d172ff01b5a33f767dc9c2a230d Nov 03 17:00:14.014883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.014843 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11339-de8381" event={"ID":"0ab08aef-2d26-41d6-b780-6c158485d372","Type":"ContainerStarted","Data":"adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2"} Nov 03 17:00:14.015076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.014899 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11339-de8381" event={"ID":"0ab08aef-2d26-41d6-b780-6c158485d372","Type":"ContainerStarted","Data":"9c24c7f7256b366dae642584b40c8125a5a67d172ff01b5a33f767dc9c2a230d"} Nov 03 17:00:14.016441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.016396 4539 generic.go:356] "Generic (PLEG): container finished" podID="c92b32cd-efeb-4455-bff8-8b38e22e6881" containerID="83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010" exitCode=137 Nov 03 17:00:14.016572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.016460 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:14.016620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.016465 4539 scope.go:122] "RemoveContainer" containerID="83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010" Nov 03 17:00:14.033271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.033242 4539 scope.go:122] "RemoveContainer" containerID="83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010" Nov 03 17:00:14.034243 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:14.034207 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010\": container with ID starting with 83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010 not found: ID does not exist" containerID="83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010" Nov 03 17:00:14.034360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.034255 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010"} err="failed to get container status \"83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010\": rpc error: code = NotFound desc = could not find container \"83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010\": container with ID starting with 83091046d05530c6d1e9ee6095c9f898c94211553ec872a5ae0ae34003481010 not found: ID does not exist" Nov 03 17:00:14.062520 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.062470 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11339-de8381" podStartSLOduration=1.062456101 podStartE2EDuration="1.062456101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:14.058452148 +0000 UTC m=+2757.751723167" watchObservedRunningTime="2026-11-03 17:00:14.062456101 +0000 UTC m=+2757.755727119" Nov 03 17:00:14.069981 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:14.069945 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11345-31a005\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="c92b32cd-efeb-4455-bff8-8b38e22e6881" pod="opendatahub/test-curl-11345-31a005" Nov 03 17:00:14.470026 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.469994 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92b32cd-efeb-4455-bff8-8b38e22e6881" path="/var/lib/kubelet/pods/c92b32cd-efeb-4455-bff8-8b38e22e6881/volumes" Nov 03 17:00:14.579098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.579065 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:14.916159 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.916129 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11339-de8381"] Nov 03 17:00:14.922037 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:14.922007 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11339-de8381"] Nov 03 17:00:15.462123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:15.462091 4539 scope.go:122] "RemoveContainer" containerID="54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73" Nov 03 17:00:15.462327 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:15.462240 4539 scope.go:122] "RemoveContainer" containerID="73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23" Nov 03 17:00:16.036877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:16.036839 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerStarted","Data":"bd1aa74a62f9785f2ded75683f7282131e1c4634d2cc90e0f810eed9c5b27be8"} Nov 03 17:00:16.039400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:16.039367 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerStarted","Data":"409d1183ec33c0bbd4df1eb917dfe046fe955f0e26629be5f184d430dba9c947"} Nov 03 17:00:16.039580 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:16.039529 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11339-de8381" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" containerName="test-curl-11339-de8381" containerID="cri-o://adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2" gracePeriod=2 Nov 03 17:00:16.123062 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:16.123004 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11339-de8381\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:16.353297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:16.346553 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:16.476898 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:16.476866 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11339-de8381\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:16.699695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:16.699595 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:17.052824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.052787 4539 generic.go:356] "Generic (PLEG): container finished" podID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerID="bd1aa74a62f9785f2ded75683f7282131e1c4634d2cc90e0f810eed9c5b27be8" exitCode=60 Nov 03 17:00:17.053318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.052865 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerDied","Data":"bd1aa74a62f9785f2ded75683f7282131e1c4634d2cc90e0f810eed9c5b27be8"} Nov 03 17:00:17.053318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.052947 4539 scope.go:122] "RemoveContainer" containerID="54c8da1f61bcd2ccb830de18ad7d1ad135ea89651dd8514ceefa3e016ef1af73" Nov 03 17:00:17.053520 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.053358 4539 scope.go:122] "RemoveContainer" containerID="bd1aa74a62f9785f2ded75683f7282131e1c4634d2cc90e0f810eed9c5b27be8" Nov 03 17:00:17.053749 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:17.053723 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj_odh-ai-gateway-infra(bae52e79-1f96-47e8-86c6-33dfc19a5974)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" Nov 03 17:00:17.055815 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.055788 4539 generic.go:356] "Generic (PLEG): container finished" podID="c951afee-0987-4757-8fe1-bc0074b3be33" containerID="409d1183ec33c0bbd4df1eb917dfe046fe955f0e26629be5f184d430dba9c947" exitCode=60 Nov 03 17:00:17.055941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.055811 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerDied","Data":"409d1183ec33c0bbd4df1eb917dfe046fe955f0e26629be5f184d430dba9c947"} Nov 03 17:00:17.056201 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.056185 4539 scope.go:122] "RemoveContainer" containerID="409d1183ec33c0bbd4df1eb917dfe046fe955f0e26629be5f184d430dba9c947" Nov 03 17:00:17.056504 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:17.056479 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6_odh-ai-gateway-infra(c951afee-0987-4757-8fe1-bc0074b3be33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" Nov 03 17:00:17.068702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.068685 4539 scope.go:122] "RemoveContainer" containerID="73192f86183eb0cad6a3022fefc42f1504f0d16d53c9c80db3546adfc30ead23" Nov 03 17:00:17.097384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.097355 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj"] Nov 03 17:00:17.145993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.145954 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6"] Nov 03 17:00:17.260794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.260760 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj"] Nov 03 17:00:17.261739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.261717 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" containerName="test-curl-11339-de8381" Nov 03 17:00:17.261845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.261741 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" containerName="test-curl-11339-de8381" Nov 03 17:00:17.261949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.261929 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" containerName="test-curl-11339-de8381" Nov 03 17:00:17.285176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.273559 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.288753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.288725 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-da409b-openshift-default-dockercfg-4js9c\"" Nov 03 17:00:17.289841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.289649 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj"] Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364200 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364278 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364326 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364394 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364444 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364518 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364605 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64nh\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364682 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.369506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.364747 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.463126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.463090 4539 scope.go:122] "RemoveContainer" containerID="1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3" Nov 03 17:00:17.470281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.464079 4539 scope.go:122] "RemoveContainer" containerID="4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498238 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498342 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64nh\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498412 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498499 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498541 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498590 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498634 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498702 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.503124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.498730 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.546672 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.546183 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.546835 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.546678 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.546916 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.546859 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.554460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.547023 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.554460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.547533 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.556465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.556394 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.558359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.558022 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.573526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.571749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:17.585252 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.585217 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.596220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.586621 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64nh\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.603077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.603038 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:17.760724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.760688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:17.775549 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.774151 4539 scope.go:122] "RemoveContainer" containerID="7807da781bdbd1d8a2572b559ddb2f84b5b692171a27ef60fec3f4d487fc1373" Nov 03 17:00:17.796289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:17.795955 4539 scope.go:122] "RemoveContainer" containerID="63a8be7fe7519c4e6b9eaaec88c97e30f68155cbd042c790525961da093f22ba" Nov 03 17:00:18.027939 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.027782 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj"] Nov 03 17:00:18.031911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.031872 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:18.032033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.031959 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:18.032033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.032003 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:18.070836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.070794 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" event={"ID":"6b9f8004-34c4-49ab-b51b-4c050c3b8b29","Type":"ContainerStarted","Data":"3f6806deb776284b7af08825c328dcd1135822b00c1e66ed1521e5a9c52116b3"} Nov 03 17:00:18.079973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.079941 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerStarted","Data":"58691efc0b1709866ff3059fef062d8c131bdb6d6c00c9fa6c98e4d39d5bcba9"} Nov 03 17:00:18.284965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.284719 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:18.382203 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.382166 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:18.402788 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.402757 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:18.426190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.426155 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca\") pod \"bae52e79-1f96-47e8-86c6-33dfc19a5974\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " Nov 03 17:00:18.426396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.426262 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c\" (UniqueName: \"kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c\") pod \"bae52e79-1f96-47e8-86c6-33dfc19a5974\" (UID: \"bae52e79-1f96-47e8-86c6-33dfc19a5974\") " Nov 03 17:00:18.426861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.426836 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca" pod "bae52e79-1f96-47e8-86c6-33dfc19a5974" (UID: "bae52e79-1f96-47e8-86c6-33dfc19a5974"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:18.429856 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.429812 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c" pod "bae52e79-1f96-47e8-86c6-33dfc19a5974" (UID: "bae52e79-1f96-47e8-86c6-33dfc19a5974"). InnerVolumeSpecName "kube-api-access-n4b2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.529847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.529476 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca\") pod \"c951afee-0987-4757-8fe1-bc0074b3be33\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " Nov 03 17:00:18.529847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.529586 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2\") pod \"0ab08aef-2d26-41d6-b780-6c158485d372\" (UID: \"0ab08aef-2d26-41d6-b780-6c158485d372\") " Nov 03 17:00:18.529847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.529652 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6\" (UniqueName: \"kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6\") pod \"c951afee-0987-4757-8fe1-bc0074b3be33\" (UID: \"c951afee-0987-4757-8fe1-bc0074b3be33\") " Nov 03 17:00:18.530504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.530259 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bae52e79-1f96-47e8-86c6-33dfc19a5974-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.530504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.530284 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4b2c\" (UniqueName: \"kubernetes.io/projected/bae52e79-1f96-47e8-86c6-33dfc19a5974-kube-api-access-n4b2c\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.547275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.537524 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca" pod "c951afee-0987-4757-8fe1-bc0074b3be33" (UID: "c951afee-0987-4757-8fe1-bc0074b3be33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:18.558157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.553373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:18.562909 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.562873 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2" pod "0ab08aef-2d26-41d6-b780-6c158485d372" (UID: "0ab08aef-2d26-41d6-b780-6c158485d372"). InnerVolumeSpecName "kube-api-access-jpmz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.564761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.564724 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6" pod "c951afee-0987-4757-8fe1-bc0074b3be33" (UID: "c951afee-0987-4757-8fe1-bc0074b3be33"). InnerVolumeSpecName "kube-api-access-lmdp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.631178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.631151 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c951afee-0987-4757-8fe1-bc0074b3be33-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.631178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.631173 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/0ab08aef-2d26-41d6-b780-6c158485d372-kube-api-access-jpmz2\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.631178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:18.631185 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdp6\" (UniqueName: \"kubernetes.io/projected/c951afee-0987-4757-8fe1-bc0074b3be33-kube-api-access-lmdp6\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.088781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.088692 4539 generic.go:356] "Generic (PLEG): container finished" podID="0ab08aef-2d26-41d6-b780-6c158485d372" containerID="adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2" exitCode=137 Nov 03 17:00:19.088781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.088743 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11339-de8381" Nov 03 17:00:19.089274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.088788 4539 scope.go:122] "RemoveContainer" containerID="adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2" Nov 03 17:00:19.090980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.090951 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" event={"ID":"6b9f8004-34c4-49ab-b51b-4c050c3b8b29","Type":"ContainerStarted","Data":"9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550"} Nov 03 17:00:19.097408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.097380 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" event={"ID":"bae52e79-1f96-47e8-86c6-33dfc19a5974","Type":"ContainerDied","Data":"9602d3872bd84344a8cd5c01d61356ea258a5ab312b3220ad14ec6f739c0a74e"} Nov 03 17:00:19.097649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.097499 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj" Nov 03 17:00:19.104582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.101847 4539 scope.go:122] "RemoveContainer" containerID="adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2" Nov 03 17:00:19.106692 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:19.106659 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2\": container with ID starting with adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2 not found: ID does not exist" containerID="adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2" Nov 03 17:00:19.106796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.106700 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2"} err="failed to get container status \"adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2\": rpc error: code = NotFound desc = could not find container \"adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2\": container with ID starting with adaa9b7eb4150ec728dd7beeb8fe96cefa87fcc8a38e0224afe490aaa879eba2 not found: ID does not exist" Nov 03 17:00:19.106796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.106728 4539 scope.go:122] "RemoveContainer" containerID="bd1aa74a62f9785f2ded75683f7282131e1c4634d2cc90e0f810eed9c5b27be8" Nov 03 17:00:19.107122 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.107100 4539 generic.go:356] "Generic (PLEG): container finished" podID="01972718-19ef-400b-a333-72872eed714b" containerID="7a2bb150107d5b622edfecb3e7e9ed4d01793027c8239b7fae464c1abcfa93b5" exitCode=60 Nov 03 17:00:19.108392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.107247 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerDied","Data":"7a2bb150107d5b622edfecb3e7e9ed4d01793027c8239b7fae464c1abcfa93b5"} Nov 03 17:00:19.108392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.107715 4539 scope.go:122] "RemoveContainer" containerID="7a2bb150107d5b622edfecb3e7e9ed4d01793027c8239b7fae464c1abcfa93b5" Nov 03 17:00:19.108392 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:19.108057 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx_odh-ai-gateway-infra(01972718-19ef-400b-a333-72872eed714b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" podUID="01972718-19ef-400b-a333-72872eed714b" Nov 03 17:00:19.109724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.109703 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" Nov 03 17:00:19.110266 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.110245 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6" event={"ID":"c951afee-0987-4757-8fe1-bc0074b3be33","Type":"ContainerDied","Data":"1b2f029444c95a1623ecd73f97e0fa96dc677cf393df35fc64a94fd01af69440"} Nov 03 17:00:19.122589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.122563 4539 generic.go:356] "Generic (PLEG): container finished" podID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerID="58691efc0b1709866ff3059fef062d8c131bdb6d6c00c9fa6c98e4d39d5bcba9" exitCode=60 Nov 03 17:00:19.122691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.122600 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerDied","Data":"58691efc0b1709866ff3059fef062d8c131bdb6d6c00c9fa6c98e4d39d5bcba9"} Nov 03 17:00:19.123014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.122993 4539 scope.go:122] "RemoveContainer" containerID="58691efc0b1709866ff3059fef062d8c131bdb6d6c00c9fa6c98e4d39d5bcba9" Nov 03 17:00:19.123359 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:19.123337 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895420-btkvm_odh-ai-gateway-infra(a8b63791-b0dc-46d7-a403-a20aee59a2a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" Nov 03 17:00:19.124694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.124676 4539 scope.go:122] "RemoveContainer" containerID="4cbb98661c1d006a7ed5ab7db9ad5efd238a7ba5d677cd413d19258ed24c78ae" Nov 03 17:00:19.148938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.148914 4539 scope.go:122] "RemoveContainer" containerID="409d1183ec33c0bbd4df1eb917dfe046fe955f0e26629be5f184d430dba9c947" Nov 03 17:00:19.150693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.150635 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podStartSLOduration=2.150618288 podStartE2EDuration="2.150618288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:19.149246221 +0000 UTC m=+2762.842517242" watchObservedRunningTime="2026-11-03 17:00:19.150618288 +0000 UTC m=+2762.843889311" Nov 03 17:00:19.164214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.164174 4539 scope.go:122] "RemoveContainer" containerID="1e9f98aa826ba941db21b1fa3eb1b6017d8dbb4325a6946353ba5debb0c152f3" Nov 03 17:00:19.187120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.187094 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm"] Nov 03 17:00:19.239537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.239509 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6"] Nov 03 17:00:19.243708 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.243676 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895420-rnrk6"] Nov 03 17:00:19.291142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.290934 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj"] Nov 03 17:00:19.296102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.295918 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895420-6zxfj"] Nov 03 17:00:19.603506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.603471 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:19.610641 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:19.610617 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:20.136231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.136210 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:20.137269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.137244 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:00:20.315894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.315868 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:20.341564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.341532 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx"] Nov 03 17:00:20.392506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.392360 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:20.454962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.454919 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7\" (UniqueName: \"kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7\") pod \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " Nov 03 17:00:20.455623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.455598 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca\") pod \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\" (UID: \"a8b63791-b0dc-46d7-a403-a20aee59a2a6\") " Nov 03 17:00:20.457168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.457124 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca" pod "a8b63791-b0dc-46d7-a403-a20aee59a2a6" (UID: "a8b63791-b0dc-46d7-a403-a20aee59a2a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:20.470366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.470323 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7" pod "a8b63791-b0dc-46d7-a403-a20aee59a2a6" (UID: "a8b63791-b0dc-46d7-a403-a20aee59a2a6"). InnerVolumeSpecName "kube-api-access-qwjk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:20.480440 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.480399 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab08aef-2d26-41d6-b780-6c158485d372" path="/var/lib/kubelet/pods/0ab08aef-2d26-41d6-b780-6c158485d372/volumes" Nov 03 17:00:20.481008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.480986 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" path="/var/lib/kubelet/pods/bae52e79-1f96-47e8-86c6-33dfc19a5974/volumes" Nov 03 17:00:20.481478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.481460 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" path="/var/lib/kubelet/pods/c951afee-0987-4757-8fe1-bc0074b3be33/volumes" Nov 03 17:00:20.558176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.558141 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b63791-b0dc-46d7-a403-a20aee59a2a6-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.558176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.558177 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwjk7\" (UniqueName: \"kubernetes.io/projected/a8b63791-b0dc-46d7-a403-a20aee59a2a6-kube-api-access-qwjk7\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.563394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.563364 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:20.591289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.591263 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:20.659869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.659782 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j\" (UniqueName: \"kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j\") pod \"01972718-19ef-400b-a333-72872eed714b\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " Nov 03 17:00:20.659869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.659861 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca\") pod \"01972718-19ef-400b-a333-72872eed714b\" (UID: \"01972718-19ef-400b-a333-72872eed714b\") " Nov 03 17:00:20.660445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.660393 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca" pod "01972718-19ef-400b-a333-72872eed714b" (UID: "01972718-19ef-400b-a333-72872eed714b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:20.662826 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.662800 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j" pod "01972718-19ef-400b-a333-72872eed714b" (UID: "01972718-19ef-400b-a333-72872eed714b"). InnerVolumeSpecName "kube-api-access-rkt4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:20.761986 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.761956 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkt4j\" (UniqueName: \"kubernetes.io/projected/01972718-19ef-400b-a333-72872eed714b-kube-api-access-rkt4j\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.761986 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:20.761984 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01972718-19ef-400b-a333-72872eed714b-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:21.147398 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.147361 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" Nov 03 17:00:21.147398 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.147387 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx" event={"ID":"01972718-19ef-400b-a333-72872eed714b","Type":"ContainerDied","Data":"3d4dbbef98400d1213aadd4d7f87703a968b46a461e01065dfa8600b1b7f307b"} Nov 03 17:00:21.148001 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.147528 4539 scope.go:122] "RemoveContainer" containerID="7a2bb150107d5b622edfecb3e7e9ed4d01793027c8239b7fae464c1abcfa93b5" Nov 03 17:00:21.149215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.149187 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" Nov 03 17:00:21.149332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.149232 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm" event={"ID":"a8b63791-b0dc-46d7-a403-a20aee59a2a6","Type":"ContainerDied","Data":"14445b6d403ad8cf1bad7112227006bbfb697b5605121205bb789b19140d0dcf"} Nov 03 17:00:21.158905 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.158883 4539 scope.go:122] "RemoveContainer" containerID="58691efc0b1709866ff3059fef062d8c131bdb6d6c00c9fa6c98e4d39d5bcba9" Nov 03 17:00:21.196094 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.196059 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm"] Nov 03 17:00:21.203715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.203681 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895420-btkvm"] Nov 03 17:00:21.243136 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.243090 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx"] Nov 03 17:00:21.256159 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.256129 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895420-hcvrx"] Nov 03 17:00:21.411573 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.411398 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:21.754048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.753972 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:21.851191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.851155 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw"] Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.851992 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852021 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852041 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852049 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852068 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852077 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852085 4539 cpu_manager.go:414] "containerMap: removing container" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852092 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852114 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852121 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852139 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852146 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852157 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852164 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852173 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852180 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852218 4539 cpu_manager.go:414] "containerMap: removing container" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852225 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852237 4539 cpu_manager.go:414] "containerMap: removing container" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.852273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852245 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852361 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852371 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852382 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852395 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852404 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae52e79-1f96-47e8-86c6-33dfc19a5974" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852415 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852445 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852454 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852462 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.853334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.852471 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:21.865935 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.865899 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:21.890248 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.890199 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e2d835-openshift-default-dockercfg-8f5zz\"" Nov 03 17:00:21.898502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:21.898456 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw"] Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.072862 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.072908 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073010 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073080 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073118 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073151 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073215 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073249 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073329 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhwt\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.073633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.073389 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.087452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.080904 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:00:22.107448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.099619 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187278 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187333 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187356 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187442 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187490 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187545 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhwt\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187587 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187669 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.188287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.187706 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.190368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.190338 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.190645 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.190615 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.192482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.192447 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.198070 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.197979 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.198664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.198637 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.203362 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.201129 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.207516 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.205582 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.220263 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.220173 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhwt\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.243284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.233390 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.282409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.282358 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:22.469506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.469415 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01972718-19ef-400b-a333-72872eed714b" path="/var/lib/kubelet/pods/01972718-19ef-400b-a333-72872eed714b/volumes" Nov 03 17:00:22.469901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.469888 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" path="/var/lib/kubelet/pods/a8b63791-b0dc-46d7-a403-a20aee59a2a6/volumes" Nov 03 17:00:22.490308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.490279 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:22.799234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.799199 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:22.826100 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.826073 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw"] Nov 03 17:00:22.827497 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:22.827471 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeacadd3_6ac7_4550_9b6b_a786de858052.slice/crio-81e1b75c0bf57135cba29a683e124b22f99da0c55b1cc29ee963be508eb13a72 WatchSource:0}: Error finding container 81e1b75c0bf57135cba29a683e124b22f99da0c55b1cc29ee963be508eb13a72: Status 404 returned error can't find the container with id 81e1b75c0bf57135cba29a683e124b22f99da0c55b1cc29ee963be508eb13a72 Nov 03 17:00:22.830847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.830814 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:22.830927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.830885 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:22.830927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:22.830915 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:23.105505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.104722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:23.221590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.221552 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" event={"ID":"feacadd3-6ac7-4550-9b6b-a786de858052","Type":"ContainerStarted","Data":"ce3bb431e76a239d18df58e13dca36b73b57341082f51def9135b36ef72c6895"} Nov 03 17:00:23.222005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.221613 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" event={"ID":"feacadd3-6ac7-4550-9b6b-a786de858052","Type":"ContainerStarted","Data":"81e1b75c0bf57135cba29a683e124b22f99da0c55b1cc29ee963be508eb13a72"} Nov 03 17:00:23.272685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.272623 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podStartSLOduration=2.272607326 podStartE2EDuration="2.272607326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:23.272586096 +0000 UTC m=+2766.965857124" watchObservedRunningTime="2026-11-03 17:00:23.272607326 +0000 UTC m=+2766.965878350" Nov 03 17:00:23.416987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.416608 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q"] Nov 03 17:00:23.416987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.416837 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" podUID="77dbf16b-0eb3-4043-a796-b3380d4ef3ce" containerName="discovery" containerID="cri-o://c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e" gracePeriod=30 Nov 03 17:00:23.490620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.490573 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:23.501501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.500333 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:23.508474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.508417 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:23.675750 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.675610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:23.900589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:23.899225 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 17:00:24.043387 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043358 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043457 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043493 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043529 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043553 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043590 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043612 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig\") pod \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\" (UID: \"77dbf16b-0eb3-4043-a796-b3380d4ef3ce\") " Nov 03 17:00:24.043947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.043834 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:24.045029 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.045001 4539 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-ca-configmap\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.046494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.046417 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:24.046625 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.046493 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "kube-api-access-wwffc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:24.046625 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.046514 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:24.046625 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.046534 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:24.047129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.047109 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:24.047233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.047216 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs" pod "77dbf16b-0eb3-4043-a796-b3380d4ef3ce" (UID: "77dbf16b-0eb3-4043-a796-b3380d4ef3ce"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:24.146237 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.146201 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwffc\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-kube-api-access-wwffc\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.146237 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.146234 4539 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-cacerts\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.146461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.146245 4539 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-local-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.146461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.146255 4539 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-csr-dns-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.146461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.146263 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.146461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.146271 4539 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77dbf16b-0eb3-4043-a796-b3380d4ef3ce-istio-kubeconfig\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:24.231174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.231139 4539 generic.go:356] "Generic (PLEG): container finished" podID="77dbf16b-0eb3-4043-a796-b3380d4ef3ce" containerID="c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e" exitCode=0 Nov 03 17:00:24.231781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.231188 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" Nov 03 17:00:24.231781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.231223 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" event={"ID":"77dbf16b-0eb3-4043-a796-b3380d4ef3ce","Type":"ContainerDied","Data":"c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e"} Nov 03 17:00:24.231781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.231293 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q" event={"ID":"77dbf16b-0eb3-4043-a796-b3380d4ef3ce","Type":"ContainerDied","Data":"92ac7ea27e3aea41ed591979762498f8e40baf95f7c2c6c152435e4feccf0ec5"} Nov 03 17:00:24.231781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.231325 4539 scope.go:122] "RemoveContainer" containerID="c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e" Nov 03 17:00:24.232098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.232075 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:24.233473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.233356 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:00:24.245230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.245211 4539 scope.go:122] "RemoveContainer" containerID="c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e" Nov 03 17:00:24.245532 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:24.245514 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e\": container with ID starting with c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e not found: ID does not exist" containerID="c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e" Nov 03 17:00:24.245589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.245541 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e"} err="failed to get container status \"c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e\": rpc error: code = NotFound desc = could not find container \"c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e\": container with ID starting with c24ec38109baa3a43265f9843d9d473a4aceb6bcf6d1b747b02569e851f81e5e not found: ID does not exist" Nov 03 17:00:24.332538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.331494 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q"] Nov 03 17:00:24.337342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.337313 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxw8q"] Nov 03 17:00:24.446033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.445998 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v"] Nov 03 17:00:24.446768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446743 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:24.446768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446765 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446797 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446802 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b63791-b0dc-46d7-a403-a20aee59a2a6" containerName="cleanup" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446824 4539 cpu_manager.go:414] "containerMap: removing container" podUID="77dbf16b-0eb3-4043-a796-b3380d4ef3ce" containerName="discovery" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446829 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77dbf16b-0eb3-4043-a796-b3380d4ef3ce" containerName="discovery" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446907 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c951afee-0987-4757-8fe1-bc0074b3be33" containerName="cleanup" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446919 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="01972718-19ef-400b-a333-72872eed714b" containerName="cleanup" Nov 03 17:00:24.447000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.446930 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="77dbf16b-0eb3-4043-a796-b3380d4ef3ce" containerName="discovery" Nov 03 17:00:24.451760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.451742 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.465535 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.465510 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-001802-openshift-default-dockercfg-skw4x\"" Nov 03 17:00:24.472200 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.472168 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77dbf16b-0eb3-4043-a796-b3380d4ef3ce" path="/var/lib/kubelet/pods/77dbf16b-0eb3-4043-a796-b3380d4ef3ce/volumes" Nov 03 17:00:24.475380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.475355 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v"] Nov 03 17:00:24.552319 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552285 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52g64\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.552534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552438 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.552534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552476 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.552697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552673 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.552804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552784 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.552862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552820 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.552922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552860 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.553013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.552990 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.553076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.553037 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.653966 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654243 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654077 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654243 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654137 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654243 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654160 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654243 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654196 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654277 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654314 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654352 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654375 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52g64\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654516 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654561 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654640 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654821 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.654938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.654916 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.657224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.657199 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.657224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.657211 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.681368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.681330 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.682083 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.682061 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52g64\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.766873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.766843 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:24.777994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:24.777953 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:25.111738 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:25.111698 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6ca69b_7f63_4326_b93c_622577097b42.slice/crio-e0c8caaa67b33885b7ed0f24682a68ce4dc7d69783444f6140d0634f9eff75a2 WatchSource:0}: Error finding container e0c8caaa67b33885b7ed0f24682a68ce4dc7d69783444f6140d0634f9eff75a2: Status 404 returned error can't find the container with id e0c8caaa67b33885b7ed0f24682a68ce4dc7d69783444f6140d0634f9eff75a2 Nov 03 17:00:25.115664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.115622 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:25.115779 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.115714 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:25.115779 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.115756 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:25.117174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.117154 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v"] Nov 03 17:00:25.244683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.244643 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" event={"ID":"1f6ca69b-7f63-4326-b93c-622577097b42","Type":"ContainerStarted","Data":"b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349"} Nov 03 17:00:25.245151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.244751 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" event={"ID":"1f6ca69b-7f63-4326-b93c-622577097b42","Type":"ContainerStarted","Data":"e0c8caaa67b33885b7ed0f24682a68ce4dc7d69783444f6140d0634f9eff75a2"} Nov 03 17:00:25.292682 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.292620 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podStartSLOduration=1.292602316 podStartE2EDuration="1.292602316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:25.287458883 +0000 UTC m=+2768.980729904" watchObservedRunningTime="2026-11-03 17:00:25.292602316 +0000 UTC m=+2768.985873340" Nov 03 17:00:25.767721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.767644 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:25.786942 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.786910 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:25.958892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:25.958859 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:26.156849 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:26.156811 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:00:26.700448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:26.699698 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:26.769782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:26.769744 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 17:00:26.769940 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:26.769798 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 17:00:27.185324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.185295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:27.568909 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.568879 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:27.730366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.730332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:27.730920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.730894 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8"] Nov 03 17:00:27.735732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.735705 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:27.744460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.744394 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8"] Nov 03 17:00:27.750003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.749975 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e2d835\"" Nov 03 17:00:27.750380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.750144 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e2d835-dockercfg-m5ld8\"" Nov 03 17:00:27.768073 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.768037 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 17:00:27.768256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.768092 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 17:00:27.776577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.776522 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c"] Nov 03 17:00:27.781762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.781730 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:27.876499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.855805 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c"] Nov 03 17:00:27.879258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.879215 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b25\" (UniqueName: \"kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25\") pod \"payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:27.879467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.879294 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:27.879467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.879454 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5t2n\" (UniqueName: \"kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n\") pod \"payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:27.879582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:27.879491 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:28.055446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.031008 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5t2n\" (UniqueName: \"kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n\") pod \"payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:28.060631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.060582 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:28.060796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.060773 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b25\" (UniqueName: \"kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25\") pod \"payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:28.060861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.060811 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:28.061654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.061627 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:28.062336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.062313 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:28.069814 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.069780 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz"] Nov 03 17:00:28.075192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.075169 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz"] Nov 03 17:00:28.075324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.075278 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.080848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.080821 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5t2n\" (UniqueName: \"kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n\") pod \"payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:28.088945 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.088913 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b25\" (UniqueName: \"kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25\") pod \"payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:28.101348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.101309 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c6f303-openshift-default-dockercfg-jfcgv\"" Nov 03 17:00:28.162529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162499 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.162688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162563 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpr7\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.162782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162708 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.162782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162772 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.162868 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162837 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.162901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162871 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.162901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.162889 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.163033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.163016 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.163122 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.163106 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.218362 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.218274 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:28.264654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264615 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.264864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264704 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.264864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264758 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpr7\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.264864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264798 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.264864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264830 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264872 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264906 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264928 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.264977 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.265071 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.265460 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.265502 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.265514 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.265831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.265786 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.267508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.267489 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.267794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.267773 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.287681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.287455 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.290025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.289980 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpr7\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.352548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.352505 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:28.443931 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.443057 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:28.552713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.552688 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c"] Nov 03 17:00:28.778450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.776221 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:28.781443 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.781400 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8"] Nov 03 17:00:28.787508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.787481 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 17:00:28.787685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.787663 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 17:00:28.795777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.795754 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz"] Nov 03 17:00:28.836377 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:28.836345 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24266884_b424_4358_9359_594731ec8628.slice/crio-e6c4f42a5cd9e1c990ba66c9d88462a3ebdb72ae0b35a6ba5b2ad777ff7a4a3c WatchSource:0}: Error finding container e6c4f42a5cd9e1c990ba66c9d88462a3ebdb72ae0b35a6ba5b2ad777ff7a4a3c: Status 404 returned error can't find the container with id e6c4f42a5cd9e1c990ba66c9d88462a3ebdb72ae0b35a6ba5b2ad777ff7a4a3c Nov 03 17:00:28.839026 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.838993 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:28.839169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.839087 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:28.839169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:28.839135 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:29.280389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.279647 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" event={"ID":"046ca71b-e9cd-4b84-acca-b294c377d0e5","Type":"ContainerStarted","Data":"784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360"} Nov 03 17:00:29.280389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.279726 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:29.280389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.279757 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" event={"ID":"046ca71b-e9cd-4b84-acca-b294c377d0e5","Type":"ContainerStarted","Data":"8a1a71a66698a0be5cb173042b9ebc7169d40b1450ce3f635088fee8c182e4e2"} Nov 03 17:00:29.281740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.281708 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" event={"ID":"24266884-b424-4358-9359-594731ec8628","Type":"ContainerStarted","Data":"ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac"} Nov 03 17:00:29.281910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.281768 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" event={"ID":"24266884-b424-4358-9359-594731ec8628","Type":"ContainerStarted","Data":"e6c4f42a5cd9e1c990ba66c9d88462a3ebdb72ae0b35a6ba5b2ad777ff7a4a3c"} Nov 03 17:00:29.283594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.283566 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" event={"ID":"08e7d664-d21b-4bb8-831b-fda26148cd6b","Type":"ContainerStarted","Data":"a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed"} Nov 03 17:00:29.283716 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.283620 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" event={"ID":"08e7d664-d21b-4bb8-831b-fda26148cd6b","Type":"ContainerStarted","Data":"b751780cde53f7f05334ecb203e8f7ab45d834709663148f0321dbf004151de4"} Nov 03 17:00:29.283716 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.283651 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:29.330250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.330172 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" podStartSLOduration=2.330151444 podStartE2EDuration="2.330151444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:29.327907635 +0000 UTC m=+2773.021178659" watchObservedRunningTime="2026-11-03 17:00:29.330151444 +0000 UTC m=+2773.023422467" Nov 03 17:00:29.381371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.381313 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podStartSLOduration=2.38129771 podStartE2EDuration="2.38129771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:29.379351591 +0000 UTC m=+2773.072622614" watchObservedRunningTime="2026-11-03 17:00:29.38129771 +0000 UTC m=+2773.074568731" Nov 03 17:00:29.427447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.422306 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" podStartSLOduration=2.422286348 podStartE2EDuration="2.422286348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:29.421837959 +0000 UTC m=+2773.115108982" watchObservedRunningTime="2026-11-03 17:00:29.422286348 +0000 UTC m=+2773.115557371" Nov 03 17:00:29.452500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.448385 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:29.452500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.448587 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:29.452500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.448656 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:29.767816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.767730 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 17:00:29.767816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:29.767797 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 17:00:30.446944 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:30.446909 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:30.447624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:30.447592 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:30.469972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:30.469939 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:30.786453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:30.781064 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 17:00:30.786453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:30.781156 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 17:00:30.946496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:30.926617 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:31.445466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:31.443930 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:31.445466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:31.443993 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:31.768672 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:31.768578 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 17:00:31.768672 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:31.768632 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 17:00:32.432117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:32.432039 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:32.444241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:32.444204 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:32.444439 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:32.444269 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:32.772545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:32.772467 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:32.773012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:32.772881 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:32.773718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:32.773697 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:00:33.443901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:33.443857 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:33.444087 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:33.443928 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:34.012102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:34.012064 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:34.444182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:34.444142 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:34.444378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:34.444203 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:34.653996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:34.653957 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p"] Nov 03 17:00:34.654191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:34.654145 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" containerID="cri-o://5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353" gracePeriod=30 Nov 03 17:00:35.265004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:35.264971 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh"] Nov 03 17:00:35.265442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:35.265206 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" containerID="cri-o://46225b336e954a4df9e7d0db994a85b7f33836599a5ed29c8469a179efc7a913" gracePeriod=30 Nov 03 17:00:35.443992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:35.443949 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 03 17:00:35.444208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:35.444000 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 03 17:00:35.498744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:35.498709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:36.449056 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:36.449025 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:36.449556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:36.449532 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:36.450296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:36.450278 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:00:36.923898 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:36.923856 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:00:36.924084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:36.923942 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:00:37.026771 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.026726 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz"] Nov 03 17:00:37.033354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.033324 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.048118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.048086 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2492c4-serving-cert\"" Nov 03 17:00:37.053308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.053007 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz"] Nov 03 17:00:37.153770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.153738 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:37.178306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.178222 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.178306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.178270 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.178556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.178535 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9nn\" (UniqueName: \"kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.178648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.178633 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.280032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.279993 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9nn\" (UniqueName: \"kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.280233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.280087 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.280233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.280148 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.280233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.280174 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.281052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.281024 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.282900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.282878 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.283196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.283177 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.302056 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.302017 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9nn\" (UniqueName: \"kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn\") pod \"maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.315250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.315217 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:37.360776 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.360738 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:37.521482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.518366 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b"] Nov 03 17:00:37.665468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.639970 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:37.797451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:37.797405 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz"] Nov 03 17:00:37.801002 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:37.799812 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb727cdc6_4114_46ef_be3a_fd5e4cd6a67a.slice/crio-3b0d4fc2ae31304e99a3ef533fa06044b2e91e486a039ac1fdb1c8338783c5be WatchSource:0}: Error finding container 3b0d4fc2ae31304e99a3ef533fa06044b2e91e486a039ac1fdb1c8338783c5be: Status 404 returned error can't find the container with id 3b0d4fc2ae31304e99a3ef533fa06044b2e91e486a039ac1fdb1c8338783c5be Nov 03 17:00:38.413597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:38.413558 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" event={"ID":"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a","Type":"ContainerStarted","Data":"7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034"} Nov 03 17:00:38.413774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:38.413629 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:38.413774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:38.413646 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" event={"ID":"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a","Type":"ContainerStarted","Data":"3b0d4fc2ae31304e99a3ef533fa06044b2e91e486a039ac1fdb1c8338783c5be"} Nov 03 17:00:38.455887 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:38.455833 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" podStartSLOduration=2.033430444 podStartE2EDuration="2.455817698s" totalImagesPullingTime="422.387254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:38.452798855 +0000 UTC m=+2782.146069877" watchObservedRunningTime="2026-11-03 17:00:38.455817698 +0000 UTC m=+2782.149088720" Nov 03 17:00:39.098037 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:39.098003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:39.254768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:39.254734 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:39.586042 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:39.586012 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:39.768522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:39.765656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:40.059270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.059241 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 17:00:40.160795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.160754 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.160827 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.160867 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.160901 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.160947 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161090 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161152 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161221 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161270 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161337 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m\") pod \"ca874712-8415-442c-892e-5bcdd0f0bffc\" (UID: \"ca874712-8415-442c-892e-5bcdd0f0bffc\") " Nov 03 17:00:40.161818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161604 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:40.163183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161951 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.163183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.161977 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca874712-8415-442c-892e-5bcdd0f0bffc-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.163183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.163037 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.163183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.163108 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.163183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.163152 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.166990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.166943 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:40.169013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.168986 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.169592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.169559 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:40.169850 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.169830 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m" pod "ca874712-8415-442c-892e-5bcdd0f0bffc" (UID: "ca874712-8415-442c-892e-5bcdd0f0bffc"). InnerVolumeSpecName "kube-api-access-8wp8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:40.263294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263264 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.263294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263292 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.263294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263301 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.263590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263310 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wp8m\" (UniqueName: \"kubernetes.io/projected/ca874712-8415-442c-892e-5bcdd0f0bffc-kube-api-access-8wp8m\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.263590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263319 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.263590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263329 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.263590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.263338 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca874712-8415-442c-892e-5bcdd0f0bffc-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.294247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.294221 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:00:40.294405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.294285 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:00:40.467507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.464028 4539 generic.go:356] "Generic (PLEG): container finished" podID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerID="5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353" exitCode=0 Nov 03 17:00:40.467507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.464129 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" Nov 03 17:00:40.472394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.472355 4539 generic.go:356] "Generic (PLEG): container finished" podID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerID="46225b336e954a4df9e7d0db994a85b7f33836599a5ed29c8469a179efc7a913" exitCode=0 Nov 03 17:00:40.473899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.473873 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" event={"ID":"ca874712-8415-442c-892e-5bcdd0f0bffc","Type":"ContainerDied","Data":"5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353"} Nov 03 17:00:40.474039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.473917 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p" event={"ID":"ca874712-8415-442c-892e-5bcdd0f0bffc","Type":"ContainerDied","Data":"206eddb5e239b98f8f0475611844aac3887459feec77f396995b73c43b5f7535"} Nov 03 17:00:40.474039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.473940 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" event={"ID":"1dc2e931-cc73-4068-ab47-f7c8a7eef053","Type":"ContainerDied","Data":"46225b336e954a4df9e7d0db994a85b7f33836599a5ed29c8469a179efc7a913"} Nov 03 17:00:40.474039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.473963 4539 scope.go:122] "RemoveContainer" containerID="5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353" Nov 03 17:00:40.497236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.496386 4539 scope.go:122] "RemoveContainer" containerID="5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353" Nov 03 17:00:40.497236 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:40.496927 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353\": container with ID starting with 5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353 not found: ID does not exist" containerID="5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353" Nov 03 17:00:40.497236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.496961 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353"} err="failed to get container status \"5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353\": rpc error: code = NotFound desc = could not find container \"5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353\": container with ID starting with 5b3b675d3f1745af29dd93ffa8ca58dfc04c269fb5cbbdd96da266eac04d2353 not found: ID does not exist" Nov 03 17:00:40.573486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.556697 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p"] Nov 03 17:00:40.573486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.562896 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-wgv6p"] Nov 03 17:00:40.664506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.664478 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 17:00:40.778791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778359 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.778791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778483 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.778791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778532 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.778791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778630 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.778791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778739 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.778791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778787 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.779261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778825 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.779261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778862 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.779261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.778919 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm\") pod \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\" (UID: \"1dc2e931-cc73-4068-ab47-f7c8a7eef053\") " Nov 03 17:00:40.780559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.780523 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.780678 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.780574 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.780811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.780767 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.781127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.781103 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.781365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.781339 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:40.783738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.783710 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:40.783917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.783897 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "kube-api-access-rjmfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:40.784192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.784171 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:40.786004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.785975 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token" pod "1dc2e931-cc73-4068-ab47-f7c8a7eef053" (UID: "1dc2e931-cc73-4068-ab47-f7c8a7eef053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:40.881444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881391 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881449 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881464 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881478 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjmfm\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-kube-api-access-rjmfm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881493 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881506 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881519 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881532 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:40.881642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:40.881543 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dc2e931-cc73-4068-ab47-f7c8a7eef053-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.050608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:41.050527 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:41.483972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:41.483897 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" Nov 03 17:00:41.484376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:41.483899 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh" event={"ID":"1dc2e931-cc73-4068-ab47-f7c8a7eef053","Type":"ContainerDied","Data":"8137222ac94d844a623ed27f8696758e5a1105c31a0615a0022e0f67cbbf5895"} Nov 03 17:00:41.484376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:41.484040 4539 scope.go:122] "RemoveContainer" containerID="46225b336e954a4df9e7d0db994a85b7f33836599a5ed29c8469a179efc7a913" Nov 03 17:00:41.548089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:41.547998 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh"] Nov 03 17:00:41.564479 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:41.562281 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-v6wqh"] Nov 03 17:00:42.471691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:42.471650 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" path="/var/lib/kubelet/pods/1dc2e931-cc73-4068-ab47-f7c8a7eef053/volumes" Nov 03 17:00:42.472378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:42.472356 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" path="/var/lib/kubelet/pods/ca874712-8415-442c-892e-5bcdd0f0bffc/volumes" Nov 03 17:00:42.836966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:42.836934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:43.145647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:43.145556 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:44.181279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:44.181246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:44.428246 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:44.428211 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:00:44.751182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:44.751149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:44.972119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:44.972088 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:45.165661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:45.165622 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:46.948285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:46.948248 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:48.586124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:48.586096 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:49.666890 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:49.666845 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg"] Nov 03 17:00:49.667404 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:49.667088 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" containerID="cri-o://4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67" gracePeriod=30 Nov 03 17:00:50.187090 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:50.187049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:50.874416 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:50.874383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:51.704112 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704076 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz"] Nov 03 17:00:51.704692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704675 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" Nov 03 17:00:51.704764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704695 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" Nov 03 17:00:51.704764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704729 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" Nov 03 17:00:51.704764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704734 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" Nov 03 17:00:51.704884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704849 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca874712-8415-442c-892e-5bcdd0f0bffc" containerName="istio-proxy" Nov 03 17:00:51.704884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.704865 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc2e931-cc73-4068-ab47-f7c8a7eef053" containerName="istio-proxy" Nov 03 17:00:51.709126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.709100 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.719949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.719912 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.720141 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.720025 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77xs\" (UniqueName: \"kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.720347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.720320 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.720414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.720381 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.732213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.732174 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-da409b-serving-cert\"" Nov 03 17:00:51.742744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.742722 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz"] Nov 03 17:00:51.821643 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.821606 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.821857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.821712 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.821857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.821736 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77xs\" (UniqueName: \"kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.821994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.821870 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.822684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.822658 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.824803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.824777 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.825229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.825208 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.825311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.825211 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:51.845233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.845201 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77xs\" (UniqueName: \"kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs\") pod \"maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:51.982657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:51.982561 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:52.043859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:52.043816 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:52.351192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:52.351161 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz"] Nov 03 17:00:52.352246 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:52.352218 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ce3998_66ad_4611_a897_911da8bb2194.slice/crio-7674b68b948ae544c4742c3c8b8405a6c2973307f270d8b8bcdb33f958d9be14 WatchSource:0}: Error finding container 7674b68b948ae544c4742c3c8b8405a6c2973307f270d8b8bcdb33f958d9be14: Status 404 returned error can't find the container with id 7674b68b948ae544c4742c3c8b8405a6c2973307f270d8b8bcdb33f958d9be14 Nov 03 17:00:52.574155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:52.574118 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" podUID="afbb4939-8cf4-4751-9128-018f943962f2" containerName="main" containerID="cri-o://04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37" gracePeriod=45 Nov 03 17:00:52.580126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:52.580090 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" event={"ID":"23ce3998-66ad-4611-a897-911da8bb2194","Type":"ContainerStarted","Data":"7674b68b948ae544c4742c3c8b8405a6c2973307f270d8b8bcdb33f958d9be14"} Nov 03 17:00:52.749322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:52.749247 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:52.950752 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:52.950498 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 17:00:53.044193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.044157 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb\" (UniqueName: \"kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb\") pod \"afbb4939-8cf4-4751-9128-018f943962f2\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " Nov 03 17:00:53.044966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.044943 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm\") pod \"afbb4939-8cf4-4751-9128-018f943962f2\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " Nov 03 17:00:53.045618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.045223 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs\") pod \"afbb4939-8cf4-4751-9128-018f943962f2\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " Nov 03 17:00:53.045997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.045971 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache\") pod \"afbb4939-8cf4-4751-9128-018f943962f2\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " Nov 03 17:00:53.046232 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.046211 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache" pod "afbb4939-8cf4-4751-9128-018f943962f2" (UID: "afbb4939-8cf4-4751-9128-018f943962f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:53.046352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.046331 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home\") pod \"afbb4939-8cf4-4751-9128-018f943962f2\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " Nov 03 17:00:53.046485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.046403 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir\") pod \"afbb4939-8cf4-4751-9128-018f943962f2\" (UID: \"afbb4939-8cf4-4751-9128-018f943962f2\") " Nov 03 17:00:53.046806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.046779 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home" pod "afbb4939-8cf4-4751-9128-018f943962f2" (UID: "afbb4939-8cf4-4751-9128-018f943962f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:53.046909 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.046804 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir" pod "afbb4939-8cf4-4751-9128-018f943962f2" (UID: "afbb4939-8cf4-4751-9128-018f943962f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:53.047170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.047154 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:53.047258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.047176 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:53.047258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.047188 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:53.048245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.048219 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs" pod "afbb4939-8cf4-4751-9128-018f943962f2" (UID: "afbb4939-8cf4-4751-9128-018f943962f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:53.050801 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.050736 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb" pod "afbb4939-8cf4-4751-9128-018f943962f2" (UID: "afbb4939-8cf4-4751-9128-018f943962f2"). InnerVolumeSpecName "kube-api-access-j8bfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:53.052139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.052110 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm" pod "afbb4939-8cf4-4751-9128-018f943962f2" (UID: "afbb4939-8cf4-4751-9128-018f943962f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:53.148564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.148523 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afbb4939-8cf4-4751-9128-018f943962f2-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:53.148564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.148561 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afbb4939-8cf4-4751-9128-018f943962f2-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:53.148810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.148580 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bfb\" (UniqueName: \"kubernetes.io/projected/afbb4939-8cf4-4751-9128-018f943962f2-kube-api-access-j8bfb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:53.148974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.148904 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc"] Nov 03 17:00:53.152749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.150041 4539 cpu_manager.go:414] "containerMap: removing container" podUID="afbb4939-8cf4-4751-9128-018f943962f2" containerName="main" Nov 03 17:00:53.152749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.150067 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbb4939-8cf4-4751-9128-018f943962f2" containerName="main" Nov 03 17:00:53.152749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.150236 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbb4939-8cf4-4751-9128-018f943962f2" containerName="main" Nov 03 17:00:53.159501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.159469 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.178449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.175277 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1f83c0-openshift-default-dockercfg-bw8dj\"" Nov 03 17:00:53.189451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.186789 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc"] Nov 03 17:00:53.250027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.249935 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.249985 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250015 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250118 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250155 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250181 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6hd\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250220 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250236 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.250417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.250277 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.261132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.261095 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:53.362311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.362272 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.374589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.374550 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.374853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.374835 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.375136 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.375117 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.375275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.375257 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.375379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.375366 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6hd\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.375565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.375549 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.375681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.375667 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.375842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.375828 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.389456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.376342 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.389710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.365993 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.389845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.386221 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.390764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.390738 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.409492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.391674 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.409773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.400700 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.412529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.412166 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.419310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.419255 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6hd\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.419622 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.419598 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.455446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.450189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:53.491707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.491663 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:53.613449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.609205 4539 generic.go:356] "Generic (PLEG): container finished" podID="afbb4939-8cf4-4751-9128-018f943962f2" containerID="04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37" exitCode=0 Nov 03 17:00:53.613449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.609322 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" event={"ID":"afbb4939-8cf4-4751-9128-018f943962f2","Type":"ContainerDied","Data":"04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37"} Nov 03 17:00:53.613449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.609359 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" event={"ID":"afbb4939-8cf4-4751-9128-018f943962f2","Type":"ContainerDied","Data":"6106056753fdcc92b6d6270e116645466adf2faf9afe88c5db23312e94a3fa23"} Nov 03 17:00:53.613449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.609384 4539 scope.go:122] "RemoveContainer" containerID="04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37" Nov 03 17:00:53.613449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.609595 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" Nov 03 17:00:53.639914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.639768 4539 scope.go:122] "RemoveContainer" containerID="04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37" Nov 03 17:00:53.640013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.639937 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" event={"ID":"23ce3998-66ad-4611-a897-911da8bb2194","Type":"ContainerStarted","Data":"ceacd04da6b58a0e689b0658487cb8c68bb6b9ec16b79c71bbe53c3b2df8e64e"} Nov 03 17:00:53.643465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.641198 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:00:53.647903 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:53.647822 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37\": container with ID starting with 04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37 not found: ID does not exist" containerID="04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37" Nov 03 17:00:53.647903 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.647863 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37"} err="failed to get container status \"04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37\": rpc error: code = NotFound desc = could not find container \"04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37\": container with ID starting with 04d5f0921bedd378622146991570761c20ee6ce93724cd99a0625a5576148f37 not found: ID does not exist" Nov 03 17:00:53.690305 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.690239 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" podStartSLOduration=2.24700217 podStartE2EDuration="2.690222475s" totalImagesPullingTime="443.220305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:53.686333009 +0000 UTC m=+2797.379604031" watchObservedRunningTime="2026-11-03 17:00:53.690222475 +0000 UTC m=+2797.383493499" Nov 03 17:00:53.727529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.727500 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b"] Nov 03 17:00:53.734387 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.734355 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b"] Nov 03 17:00:53.841019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.840942 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc"] Nov 03 17:00:53.841790 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:53.841755 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e9ff44_d087_40e5_b105_07b8c01b0686.slice/crio-900c9e5106e6ed2efa7690611702c3704f04eb1b0cd84b9709af900c4dd45242 WatchSource:0}: Error finding container 900c9e5106e6ed2efa7690611702c3704f04eb1b0cd84b9709af900c4dd45242: Status 404 returned error can't find the container with id 900c9e5106e6ed2efa7690611702c3704f04eb1b0cd84b9709af900c4dd45242 Nov 03 17:00:53.850460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.850081 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:53.850460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.850174 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:53.850460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.850220 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:00:53.869527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.869494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:53.870725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:53.870689 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-c6f303/auth-test-model-c6f303-kserve-6779479b77-x466b" podUID="afbb4939-8cf4-4751-9128-018f943962f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:00:54.471966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:54.471931 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbb4939-8cf4-4751-9128-018f943962f2" path="/var/lib/kubelet/pods/afbb4939-8cf4-4751-9128-018f943962f2/volumes" Nov 03 17:00:54.649999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:54.649966 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" event={"ID":"74e9ff44-d087-40e5-b105-07b8c01b0686","Type":"ContainerStarted","Data":"b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa"} Nov 03 17:00:54.650193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:54.650026 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" event={"ID":"74e9ff44-d087-40e5-b105-07b8c01b0686","Type":"ContainerStarted","Data":"900c9e5106e6ed2efa7690611702c3704f04eb1b0cd84b9709af900c4dd45242"} Nov 03 17:00:54.700130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:54.700057 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podStartSLOduration=1.700036155 podStartE2EDuration="1.700036155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:54.698536615 +0000 UTC m=+2798.391807638" watchObservedRunningTime="2026-11-03 17:00:54.700036155 +0000 UTC m=+2798.393307178" Nov 03 17:00:54.999907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:54.999876 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 17:00:55.113394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113294 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.113394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113353 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.113706 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113398 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.113706 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113530 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.113706 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113603 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.113706 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113670 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.113706 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113699 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.114208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113728 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.114208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113745 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.114208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113778 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs\") pod \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\" (UID: \"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a\") " Nov 03 17:00:55.114208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113816 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:55.114208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113941 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.114208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.113956 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.114566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.114237 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.114566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.114372 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.114566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.114393 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.114566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.114408 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.114566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.114440 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.114566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.114453 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.116219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.116193 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:55.116336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.116232 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.116830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.116802 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:55.117994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.117968 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp" pod "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" (UID: "aaf7c0ad-212d-46ed-9ce2-e874e3213f2a"). InnerVolumeSpecName "kube-api-access-7b7mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:55.216113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.216083 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.216113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.216111 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.216113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.216123 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.216352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.216135 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b7mp\" (UniqueName: \"kubernetes.io/projected/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a-kube-api-access-7b7mp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.320515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.320476 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn"] Nov 03 17:00:55.321038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.321022 4539 cpu_manager.go:414] "containerMap: removing container" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" Nov 03 17:00:55.321038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.321038 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" Nov 03 17:00:55.321188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.321155 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerName="istio-proxy" Nov 03 17:00:55.326590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.325302 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.339120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.339091 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e2d835-serving-cert\"" Nov 03 17:00:55.349262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.349217 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn"] Nov 03 17:00:55.417911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.417829 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wpt\" (UniqueName: \"kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.417911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.417869 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.418109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.418043 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.418109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.418085 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.442134 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.442103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:55.492755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.492725 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:55.498956 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.498932 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:55.519961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.519929 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wpt\" (UniqueName: \"kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.520085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.519977 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.520170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.520155 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.520340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.520316 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.520879 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.520855 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.522839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.522813 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.522926 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.522822 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.547526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.543197 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wpt\" (UniqueName: \"kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt\") pod \"maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.595253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.595191 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:55.659483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.659450 4539 generic.go:356] "Generic (PLEG): container finished" podID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" containerID="4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67" exitCode=0 Nov 03 17:00:55.659483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.659470 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" event={"ID":"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a","Type":"ContainerDied","Data":"4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67"} Nov 03 17:00:55.659744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.659535 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" Nov 03 17:00:55.659744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.659549 4539 scope.go:122] "RemoveContainer" containerID="4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67" Nov 03 17:00:55.659744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.659536 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg" event={"ID":"aaf7c0ad-212d-46ed-9ce2-e874e3213f2a","Type":"ContainerDied","Data":"299fdafd9d770e68f4a44f39a6e53568f4b0adcd6c5d5cd53b365365eae93a1f"} Nov 03 17:00:55.660352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.660331 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:55.661155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.660814 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:55.661447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.661403 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:00:55.676734 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.676712 4539 scope.go:122] "RemoveContainer" containerID="4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67" Nov 03 17:00:55.677019 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:55.677002 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67\": container with ID starting with 4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67 not found: ID does not exist" containerID="4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67" Nov 03 17:00:55.677074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.677028 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67"} err="failed to get container status \"4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67\": rpc error: code = NotFound desc = could not find container \"4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67\": container with ID starting with 4ccf48011d8266e042ad708c0dd811df5ae8826c81530c87020d0c39277b1d67 not found: ID does not exist" Nov 03 17:00:55.709691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.709657 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg"] Nov 03 17:00:55.716472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:55.716123 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-zgrhg"] Nov 03 17:00:56.004200 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:00:56.004165 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc59b182_642d_4931_8dad_2a1885caf806.slice/crio-8f74c7fa3ebe3b834d1ee1de0703fffca48f84e59341a33c09d323e2762bbb9a WatchSource:0}: Error finding container 8f74c7fa3ebe3b834d1ee1de0703fffca48f84e59341a33c09d323e2762bbb9a: Status 404 returned error can't find the container with id 8f74c7fa3ebe3b834d1ee1de0703fffca48f84e59341a33c09d323e2762bbb9a Nov 03 17:00:56.006607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:56.006580 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn"] Nov 03 17:00:56.156985 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:00:56.156942 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:00:56.474172 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:56.474138 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf7c0ad-212d-46ed-9ce2-e874e3213f2a" path="/var/lib/kubelet/pods/aaf7c0ad-212d-46ed-9ce2-e874e3213f2a/volumes" Nov 03 17:00:56.675703 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:56.675624 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" event={"ID":"fc59b182-642d-4931-8dad-2a1885caf806","Type":"ContainerStarted","Data":"1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8"} Nov 03 17:00:56.675703 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:56.675676 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" event={"ID":"fc59b182-642d-4931-8dad-2a1885caf806","Type":"ContainerStarted","Data":"8f74c7fa3ebe3b834d1ee1de0703fffca48f84e59341a33c09d323e2762bbb9a"} Nov 03 17:00:56.676550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:56.676532 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:00:56.717219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:56.717154 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" podStartSLOduration=1.424773453 podStartE2EDuration="1.717130271s" totalImagesPullingTime="292.356818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:56.716292622 +0000 UTC m=+2800.409563646" watchObservedRunningTime="2026-11-03 17:00:56.717130271 +0000 UTC m=+2800.410401294" Nov 03 17:00:57.403885 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:57.403846 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:57.584549 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:57.584512 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:58.828564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:58.828528 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:59.225596 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:59.225494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:00:59.376282 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:00:59.376250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:00.666843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:00.666813 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:01:01.116356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:01.116195 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:02.886775 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:02.886729 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:03.698061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:03.698035 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:01:04.469080 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:04.469033 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:05.992249 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:05.992207 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:07.670215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:07.670186 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:09.285658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:09.285628 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:11.380256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:11.380208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:13.421580 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:13.421510 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:13.619392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:13.617052 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:14.466191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:14.466148 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:14.527757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:14.527696 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:14.627735 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:14.627698 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:14.628126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:14.628007 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:14.679592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:14.679541 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:14.850256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:14.850212 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:15.096817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:15.096782 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:15.350771 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:15.306501 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:15.485664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:15.485607 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:15.939850 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:15.939815 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl"] Nov 03 17:01:15.978363 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:15.978261 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:16.084445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.054760 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.164448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.142186 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6046c2-openshift-default-dockercfg-47pst\"" Nov 03 17:01:16.200971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.167374 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl"] Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218081 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218160 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218207 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpn62\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218357 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218410 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218519 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218606 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218660 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.260451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.218723 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.494900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.494806 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.495547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.495517 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.511448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.495686 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520291 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn62\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520509 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520566 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520612 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520661 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520712 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.521091 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.514973 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.520233 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.521680 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.521919 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.532457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.522558 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.603710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.598785 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.603710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.599106 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.643107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.642302 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn62\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.660906 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.660876 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6046c2-openshift-default-dockercfg-47pst\"" Nov 03 17:01:16.676663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.676629 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:16.682587 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.677306 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:16.911449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:16.896968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:17.350066 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:17.349978 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl"] Nov 03 17:01:17.360473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:17.360406 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:17.360668 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:17.360542 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:17.360668 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:17.360627 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:18.217106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:18.215876 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" event={"ID":"c641e1c4-3301-4c83-85a1-f17596684ea7","Type":"ContainerStarted","Data":"3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178"} Nov 03 17:01:18.217106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:18.215940 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" event={"ID":"c641e1c4-3301-4c83-85a1-f17596684ea7","Type":"ContainerStarted","Data":"66b91831f8803948aa69b865fada171ac98f3b79e31f8c43461f34f1ed756e49"} Nov 03 17:01:18.271401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:18.271332 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podStartSLOduration=3.271315324 podStartE2EDuration="3.271315324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:18.269666426 +0000 UTC m=+2821.962937463" watchObservedRunningTime="2026-11-03 17:01:18.271315324 +0000 UTC m=+2821.964586346" Nov 03 17:01:18.517126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:18.516996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:18.677645 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:18.677611 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:18.685283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:18.685258 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:19.228293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:19.228238 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:19.229488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:19.229172 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:01:20.195336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:20.195254 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:20.791552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:20.791504 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:21.113923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:21.113808 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:21.178638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:21.178587 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:01:21.178861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:21.178669 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:21.935655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:21.935560 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:01:21.935655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:21.935634 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.137273 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z"] Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.160861 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.181111 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z"] Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.257996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266494 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266541 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266682 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266814 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266850 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266909 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.266999 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.267087 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.267117 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5br\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368055 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368096 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368132 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368194 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368216 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368243 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368346 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368382 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368463 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5br\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368552 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368564 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368675 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.368926 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.369218 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.370811 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.371345 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.391130 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.391238 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5br\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.414987 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.517229 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.587856 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:22.884523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.744219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:22.968870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.968843 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z"] Nov 03 17:01:22.971686 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:01:22.971660 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb17265_a6f5_449b_b8b9_c10f2d8aadcc.slice/crio-5910112388a6e7daed3e1b91d86673222f150a710bd858236f75a873711e6021 WatchSource:0}: Error finding container 5910112388a6e7daed3e1b91d86673222f150a710bd858236f75a873711e6021: Status 404 returned error can't find the container with id 5910112388a6e7daed3e1b91d86673222f150a710bd858236f75a873711e6021 Nov 03 17:01:22.974590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.974561 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:22.974670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.974631 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:22.974670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:22.974664 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.267265 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" event={"ID":"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc","Type":"ContainerStarted","Data":"e380d1ca6a06d5c634b21cd6ae86517f88986b2b92ff1dd3f35a8f5cc7facf2f"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.267315 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" event={"ID":"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc","Type":"ContainerStarted","Data":"5910112388a6e7daed3e1b91d86673222f150a710bd858236f75a873711e6021"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.269152 4539 generic.go:356] "Generic (PLEG): container finished" podID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerID="9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550" exitCode=0 Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.269222 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" event={"ID":"6b9f8004-34c4-49ab-b51b-4c050c3b8b29","Type":"ContainerDied","Data":"9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.269609 4539 scope.go:122] "RemoveContainer" containerID="9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550" Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.271208 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.271281 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.271313 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.318853 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podStartSLOduration=1.318834369 podStartE2EDuration="1.318834369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:23.318079573 +0000 UTC m=+2827.011350595" watchObservedRunningTime="2026-11-03 17:01:23.318834369 +0000 UTC m=+2827.012105392" Nov 03 17:01:23.525368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:23.518212 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:24.108140 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.108096 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:24.245451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.242558 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:24.245451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.242625 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:24.288515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.288484 4539 generic.go:356] "Generic (PLEG): container finished" podID="feacadd3-6ac7-4550-9b6b-a786de858052" containerID="ce3bb431e76a239d18df58e13dca36b73b57341082f51def9135b36ef72c6895" exitCode=0 Nov 03 17:01:24.288730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.288613 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" event={"ID":"feacadd3-6ac7-4550-9b6b-a786de858052","Type":"ContainerDied","Data":"ce3bb431e76a239d18df58e13dca36b73b57341082f51def9135b36ef72c6895"} Nov 03 17:01:24.289194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.289171 4539 scope.go:122] "RemoveContainer" containerID="ce3bb431e76a239d18df58e13dca36b73b57341082f51def9135b36ef72c6895" Nov 03 17:01:24.291331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.291298 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:24.291478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.291389 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:24.291584 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.291481 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:24.292352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.292010 4539 generic.go:356] "Generic (PLEG): container finished" podID="80150068-7b02-4d59-84f7-1e23530dd995" containerID="9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e" exitCode=0 Nov 03 17:01:24.292352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.292099 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" event={"ID":"80150068-7b02-4d59-84f7-1e23530dd995","Type":"ContainerDied","Data":"9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e"} Nov 03 17:01:24.292488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.292445 4539 scope.go:122] "RemoveContainer" containerID="9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e" Nov 03 17:01:24.294435 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.294058 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:24.294435 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.294116 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:24.294435 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.294147 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:24.296836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.296771 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" event={"ID":"6b9f8004-34c4-49ab-b51b-4c050c3b8b29","Type":"ContainerStarted","Data":"06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9"} Nov 03 17:01:24.531687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.531652 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 17:01:24.531881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.531715 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 17:01:24.619447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.607803 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:01:24.619447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.607926 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:24.619447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:24.607983 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:25.311383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.310312 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" event={"ID":"feacadd3-6ac7-4550-9b6b-a786de858052","Type":"ContainerStarted","Data":"a83f8694195e00bce48b9b51e3841cecad5413f5d90545951477e70be62b6795"} Nov 03 17:01:25.316644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.315494 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" event={"ID":"80150068-7b02-4d59-84f7-1e23530dd995","Type":"ContainerStarted","Data":"8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd"} Nov 03 17:01:25.491837 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.491804 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:01:25.492115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.492003 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:25.492254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.492132 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:25.522916 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.522881 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 17:01:25.523259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.523218 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 17:01:25.568060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.567935 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:01:25.568500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.568478 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:25.568698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.568651 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:25.604457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.603723 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:25.604457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.603775 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:25.927408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:25.927312 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:26.295473 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:01:26.294162 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:01:26.501453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.500747 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:26.501453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.500806 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:26.532013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.531975 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 17:01:26.532285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.532258 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 17:01:26.566304 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.566218 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:26.566596 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.566568 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:26.617466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.617332 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:26.617466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:26.617385 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:27.460591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.460543 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 17:01:27.498455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.498098 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:27.498455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.498187 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:27.539867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.539815 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 17:01:27.540528 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.540491 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 17:01:27.584569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.584534 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:27.584852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.584825 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:27.615563 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.615529 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:27.620322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.620287 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:27.731586 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.731446 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:27.847561 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:27.847520 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:28.282450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.281513 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:28.502296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.485679 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:28.502296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.491406 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:28.502296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.491472 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:28.535011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.534272 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 17:01:28.535011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.534331 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 17:01:28.563448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.537948 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 17:01:28.587446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.577247 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:28.587446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.577331 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:28.589532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.589495 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:28.643464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.617313 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:28.643464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.617382 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:28.671476 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.665525 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:28.920503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.889334 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:28.920503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.889827 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 17:01:28.920858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:28.920818 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 03 17:01:29.314550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.314510 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:29.315286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.315255 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 17:01:29.504715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.504668 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:29.567612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.557973 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:29.567612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.558030 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:29.567612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.563964 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 17:01:29.567612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.564013 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 17:01:29.638873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.597699 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:29.638873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.597757 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:29.638873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.598680 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:29.638873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.598987 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 17:01:29.638873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.604639 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:29.638873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.604678 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:29.814574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.786548 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:29.814574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.787320 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:29.842605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.827568 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 17:01:29.884451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.880549 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 03 17:01:29.968364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:29.911640 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:30.094651 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.084370 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 17:01:30.375978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.371555 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:30.494994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.494560 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:30.494994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.494622 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:30.512404 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.509539 4539 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:30.512404 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.511897 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:30.512404 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.512134 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 17:01:30.587451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.585829 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:30.587451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.585887 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:30.625064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.624345 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 03 17:01:30.625064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.624402 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 03 17:01:30.749530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.736752 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:30.749530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.740781 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:30.771448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.764789 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:01:30.789566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.789525 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:30.907455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.898785 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 17:01:30.907763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.907741 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 17:01:30.989763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.989735 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 17:01:30.997752 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:30.997726 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 17:01:31.505298 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.505260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:31.506044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.506020 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:31.506244 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.506221 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:31.570859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.569843 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:31.570859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.569905 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:31.667210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.667169 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:31.832868 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.832753 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:31.853471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.851930 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:01:31.853471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.853383 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:01:31.864922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:31.864709 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:01:32.492451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:32.491804 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 17:01:32.492451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:32.491866 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 17:01:32.569971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:32.569933 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 03 17:01:32.570289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:32.570229 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 03 17:01:33.465592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.465558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:33.495903 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.495872 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:01:33.496266 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.496233 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:01:33.497013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.496982 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:01:33.572117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.572081 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:01:33.572391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.572376 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:01:33.573175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:33.573157 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:01:34.700866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:34.700835 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:34.893290 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:34.893251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:35.152831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.152793 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:35.246345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.246314 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk"] Nov 03 17:01:35.252862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.252840 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.286234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.282719 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e0e63d-openshift-default-dockercfg-72z97\"" Nov 03 17:01:35.308998 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.308967 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk"] Nov 03 17:01:35.392521 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.392488 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:35.444811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.444612 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.444811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.444669 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.444811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.444708 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.444811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.444791 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.445173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.444888 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmg9\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.445173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.444996 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.445173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.445027 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.445173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.445083 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.445173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.445135 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.548817 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.548903 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.548940 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.548966 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.549001 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.549094 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.549203 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmg9\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.549332 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.552445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.549366 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.554010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.553977 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.568445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.554840 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.568758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.555968 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.568874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.555991 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.568966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.565173 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.573513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.567376 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.579075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.579039 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.599976 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.599944 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmg9\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.600830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.600806 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:35.648491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.647626 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:35.890861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:35.890828 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:36.231381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:36.231285 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk"] Nov 03 17:01:36.243443 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:01:36.243393 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6fa8fa_8315_4726_9f8e_0a54a7fb33d3.slice/crio-98562a362040a2b5b86c79f1be48b853c386dbad65f4bb38397dd5cb98e21603 WatchSource:0}: Error finding container 98562a362040a2b5b86c79f1be48b853c386dbad65f4bb38397dd5cb98e21603: Status 404 returned error can't find the container with id 98562a362040a2b5b86c79f1be48b853c386dbad65f4bb38397dd5cb98e21603 Nov 03 17:01:36.246921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:36.246884 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:36.247015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:36.246977 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:36.247065 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:36.247019 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:36.760370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:36.760330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:37.112002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:37.111962 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" event={"ID":"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3","Type":"ContainerStarted","Data":"6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d"} Nov 03 17:01:37.112547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:37.112024 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" event={"ID":"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3","Type":"ContainerStarted","Data":"98562a362040a2b5b86c79f1be48b853c386dbad65f4bb38397dd5cb98e21603"} Nov 03 17:01:37.170442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:37.170362 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podStartSLOduration=2.170340935 podStartE2EDuration="2.170340935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:37.163994533 +0000 UTC m=+2840.857265562" watchObservedRunningTime="2026-11-03 17:01:37.170340935 +0000 UTC m=+2840.863611957" Nov 03 17:01:37.891365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:37.891326 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:37.897344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:37.897320 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:38.120520 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:38.120491 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:38.121682 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:38.121659 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:01:38.475962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:38.475929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:39.574183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:39.574152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:39.753844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:39.753810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:39.928523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:39.928410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:40.281891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:40.281859 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:40.920264 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:40.920230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:41.299870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:41.299835 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:41.466125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:41.466091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:41.718378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:41.715761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:42.514052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:42.514018 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:43.293200 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:43.293168 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:43.473655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:43.473622 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:44.123658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:44.123616 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:45.792181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:45.792152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:47.488150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:47.488116 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:49.438095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:49.438060 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:51.030929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:51.030876 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:52.663154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:52.663116 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:54.468031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:54.467993 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:55.487523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:55.487491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:55.844744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:55.844711 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48"] Nov 03 17:01:55.855047 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:55.855023 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:55.879079 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:55.879041 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ca9d7c-openshift-default-dockercfg-fgrm2\"" Nov 03 17:01:55.893692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:55.893664 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48"] Nov 03 17:01:55.948603 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:55.948530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:56.008975 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.008922 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2ql\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.009354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.009330 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.009539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.009520 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.009718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.009703 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.009897 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.009880 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.010099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.010084 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.010213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.010199 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.010322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.010307 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.010456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.010440 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.120407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.120312 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2ql\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.120755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.120728 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.120912 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.120898 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.121061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.121047 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.121201 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.121186 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.121396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.121380 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.121579 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.121562 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.121704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.121688 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.121819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.121806 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.131454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.126452 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.131454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.126736 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.154480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.142751 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.154480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.143313 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.154480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.143593 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.154480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.149730 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:56.154480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.153135 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.154480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.153742 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.158393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.158346 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2ql\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.177445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.176986 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.191458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.190752 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:56.265281 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:01:56.265207 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:01:56.539657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.539619 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v"] Nov 03 17:01:56.545071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.545049 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.571522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.571493 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6046c2-serving-cert\"" Nov 03 17:01:56.580984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.580948 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v"] Nov 03 17:01:56.626108 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.626074 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48"] Nov 03 17:01:56.641138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.641105 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqn4\" (UniqueName: \"kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.641325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.641241 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.641398 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.641380 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.641483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.641412 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.647004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.646960 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:56.647271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.647225 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:56.647449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.647407 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:01:56.698015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.697986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:56.752104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.749376 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.758928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.758685 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.762518 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.749548 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.762686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.762566 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.776353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.762803 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqn4\" (UniqueName: \"kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.783326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.783249 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.784145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.784089 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.800391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.800320 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqn4\" (UniqueName: \"kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4\") pod \"maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:56.851297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.851230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:56.877827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:56.877787 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:57.019521 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:57.019343 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:57.270469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:57.270395 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v"] Nov 03 17:01:57.314478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:57.310221 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" event={"ID":"266557c9-7986-414b-8dbe-d373417c4b12","Type":"ContainerStarted","Data":"0a9593ee0a2f34a233b68b242d844ee8ba69f2abc0d825274a8ff09572816180"} Nov 03 17:01:57.314478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:57.312624 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" event={"ID":"f7b13c5a-b79f-40ef-98bc-b0606db9772e","Type":"ContainerStarted","Data":"834af19cd358133b3588d89c8166ae73ad57a42726fd3e397191dd79c4a81d17"} Nov 03 17:01:57.314478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:57.312684 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" event={"ID":"f7b13c5a-b79f-40ef-98bc-b0606db9772e","Type":"ContainerStarted","Data":"6d6f67ed4b2c037bd6369894b30b1fb1456b841c5ed9f3c62930b4ad0a13008b"} Nov 03 17:01:57.370934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:57.370857 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podStartSLOduration=2.370836417 podStartE2EDuration="2.370836417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:57.364181825 +0000 UTC m=+2861.057452847" watchObservedRunningTime="2026-11-03 17:01:57.370836417 +0000 UTC m=+2861.064107438" Nov 03 17:01:58.193794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.193760 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:58.199740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.199715 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:58.322890 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.322854 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" event={"ID":"266557c9-7986-414b-8dbe-d373417c4b12","Type":"ContainerStarted","Data":"be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441"} Nov 03 17:01:58.323086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.322947 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:01:58.323413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.323387 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:58.324491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.324469 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:01:58.365034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.364961 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" podStartSLOduration=1.96145561 podStartE2EDuration="2.364947749s" totalImagesPullingTime="403.492139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:58.363699311 +0000 UTC m=+2862.056970332" watchObservedRunningTime="2026-11-03 17:01:58.364947749 +0000 UTC m=+2862.058218793" Nov 03 17:01:58.495453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:58.495363 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:59.766552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:59.766517 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:01:59.921922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:01:59.921856 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:00.524912 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:00.524884 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:01.104732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:01.104694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:01.513168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:01.513084 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:01.672938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:01.672911 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:02.406589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:02.406555 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:04.181795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:04.181752 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:04.336758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:04.336730 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:02:05.854970 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:05.853498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:07.452936 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:07.452885 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:08.987203 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:08.987165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:10.750018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:10.749979 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:12.420474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:12.420438 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:13.478839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:13.478805 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:14.017263 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:14.017229 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:15.186540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.186498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:15.525109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.525075 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84"] Nov 03 17:02:15.562309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.562281 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.583509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.583481 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84"] Nov 03 17:02:15.587741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.587602 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3a0e27-serving-cert\"" Nov 03 17:02:15.590535 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.590505 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.590794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.590774 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.591022 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.591003 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9nd\" (UniqueName: \"kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.591352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.591331 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.653964 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.653914 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:15.693244 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.693209 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9nd\" (UniqueName: \"kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.693687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.693651 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.693862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.693847 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.694007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.693994 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.695341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.695311 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.695917 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:02:15.695885 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-serving-cert: secret "maas-api-e2e-extmod-a-3a0e27-serving-cert" not found Nov 03 17:02:15.696086 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:02:15.696075 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls podName:b5311ef6-beae-4a0f-a463-951e90ff1f6f nodeName:}" failed. No retries permitted until 2026-11-03 17:02:16.196055156 +0000 UTC m=+2879.889326162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls") pod "maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" (UID: "b5311ef6-beae-4a0f-a463-951e90ff1f6f") : secret "maas-api-e2e-extmod-a-3a0e27-serving-cert" not found Nov 03 17:02:15.744447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.718380 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.744447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.742276 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9nd\" (UniqueName: \"kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:15.836619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.836520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:15.991965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:15.991932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:16.137033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:16.136939 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:16.214371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:16.214329 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:16.220049 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:16.220017 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:16.489899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:16.489697 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:16.873577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:16.873265 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84"] Nov 03 17:02:16.873990 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:02:16.873942 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5311ef6_beae_4a0f_a463_951e90ff1f6f.slice/crio-1326a25033bd6ee4e44f016a3487fa058300b59ed145731ccc488d47dc427a2e WatchSource:0}: Error finding container 1326a25033bd6ee4e44f016a3487fa058300b59ed145731ccc488d47dc427a2e: Status 404 returned error can't find the container with id 1326a25033bd6ee4e44f016a3487fa058300b59ed145731ccc488d47dc427a2e Nov 03 17:02:17.478272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.478186 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd"] Nov 03 17:02:17.520885 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.520846 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.521533 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.521509 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd"] Nov 03 17:02:17.537483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.533606 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" event={"ID":"b5311ef6-beae-4a0f-a463-951e90ff1f6f","Type":"ContainerStarted","Data":"8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883"} Nov 03 17:02:17.537483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.533665 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" event={"ID":"b5311ef6-beae-4a0f-a463-951e90ff1f6f","Type":"ContainerStarted","Data":"1326a25033bd6ee4e44f016a3487fa058300b59ed145731ccc488d47dc427a2e"} Nov 03 17:02:17.537483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.534492 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:17.591487 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.588087 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e0e63d-serving-cert\"" Nov 03 17:02:17.608514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.607178 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:17.705257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.705221 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfll9\" (UniqueName: \"kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.705599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.705538 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.705923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.705901 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.706079 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.706063 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.787721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.787673 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:17.807408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.807371 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.808566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.808537 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.809220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.808822 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfll9\" (UniqueName: \"kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.809220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.808869 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.809686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.809658 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.812120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.812098 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.821986 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.821960 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.868236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.868201 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfll9\" (UniqueName: \"kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9\") pod \"maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:17.965383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:17.965346 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:18.003871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:18.003824 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:18.280852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:18.280806 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" podStartSLOduration=2.945994522 podStartE2EDuration="3.280790354s" totalImagesPullingTime="334.795832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:17.659732207 +0000 UTC m=+2881.353003234" watchObservedRunningTime="2026-11-03 17:02:18.280790354 +0000 UTC m=+2881.974061376" Nov 03 17:02:18.282459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:18.282418 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd"] Nov 03 17:02:18.283403 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:02:18.283273 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f63229_5b65_4992_ba0c_69baf3a814a4.slice/crio-f762d70b9ba033a976200633de5cb9a5546eb42765a934c7ae25c8a626d01c86 WatchSource:0}: Error finding container f762d70b9ba033a976200633de5cb9a5546eb42765a934c7ae25c8a626d01c86: Status 404 returned error can't find the container with id f762d70b9ba033a976200633de5cb9a5546eb42765a934c7ae25c8a626d01c86 Nov 03 17:02:18.669633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:18.669552 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" event={"ID":"d3f63229-5b65-4992-ba0c-69baf3a814a4","Type":"ContainerStarted","Data":"f762d70b9ba033a976200633de5cb9a5546eb42765a934c7ae25c8a626d01c86"} Nov 03 17:02:19.165934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:19.165899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:19.333452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:19.332941 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:19.679658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:19.679621 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" event={"ID":"d3f63229-5b65-4992-ba0c-69baf3a814a4","Type":"ContainerStarted","Data":"17369f2360a458b78cab3c15b6c76160f7c00e8656f8d205c7d4df20bec5e875"} Nov 03 17:02:19.680313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:19.680286 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:19.729180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:19.723965 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" podStartSLOduration=2.285749584 podStartE2EDuration="2.723943493s" totalImagesPullingTime="438.193909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:19.721816654 +0000 UTC m=+2883.415087678" watchObservedRunningTime="2026-11-03 17:02:19.723943493 +0000 UTC m=+2883.417214518" Nov 03 17:02:19.848647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:19.846336 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:20.521452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:20.521094 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:20.934253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:20.934221 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:21.169677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:21.169589 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:21.913657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:21.913456 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:23.594839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.594805 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:23.736820 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.736789 4539 generic.go:356] "Generic (PLEG): container finished" podID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerID="b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa" exitCode=0 Nov 03 17:02:23.737022 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.736865 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" event={"ID":"74e9ff44-d087-40e5-b105-07b8c01b0686","Type":"ContainerDied","Data":"b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa"} Nov 03 17:02:23.737340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.737323 4539 scope.go:122] "RemoveContainer" containerID="b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa" Nov 03 17:02:23.739441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.739393 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:23.739574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.739499 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:23.739646 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:23.739576 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:24.686822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:24.686791 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:02:24.747601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:24.747567 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" event={"ID":"74e9ff44-d087-40e5-b105-07b8c01b0686","Type":"ContainerStarted","Data":"ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0"} Nov 03 17:02:25.396125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:25.396089 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:25.492740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:25.492709 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:02:25.493953 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:25.493913 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 17:02:25.494114 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:25.493971 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 17:02:25.660338 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:25.660245 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:02:26.160823 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:02:26.160777 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:02:26.492729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:26.492644 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 17:02:26.492915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:26.492727 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 17:02:26.729650 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:26.729613 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:02:27.080057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:27.080024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:27.493852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:27.493772 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 17:02:27.493852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:27.493833 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 17:02:28.492915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:28.492883 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 17:02:28.493099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:28.492945 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 17:02:28.684917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:28.684873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:29.493731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.493634 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 03 17:02:29.493731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.493694 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 03 17:02:29.792583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.792548 4539 generic.go:356] "Generic (PLEG): container finished" podID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerID="3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178" exitCode=0 Nov 03 17:02:29.793048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.792632 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" event={"ID":"c641e1c4-3301-4c83-85a1-f17596684ea7","Type":"ContainerDied","Data":"3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178"} Nov 03 17:02:29.793129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.793094 4539 scope.go:122] "RemoveContainer" containerID="3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178" Nov 03 17:02:29.794894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.794871 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:29.794949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.794937 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:29.794984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:29.794972 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:30.337510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:30.337409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:30.497436 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:30.497404 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:02:30.498612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:30.498591 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:02:30.802519 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:30.802476 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" event={"ID":"c641e1c4-3301-4c83-85a1-f17596684ea7","Type":"ContainerStarted","Data":"b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1"} Nov 03 17:02:31.677834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:31.677804 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:02:31.678685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:31.678662 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 17:02:31.678809 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:31.678707 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 17:02:32.086271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:32.086209 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:32.678714 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:32.678677 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 17:02:32.678914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:32.678735 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 17:02:33.678091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:33.678055 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 17:02:33.678539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:33.678110 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 17:02:33.716834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:33.716802 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:33.865011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:33.864980 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:34.228606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:34.228572 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:02:34.678357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:34.678322 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 17:02:34.678799 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:34.678374 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 17:02:35.389540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:35.389505 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:35.678157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:35.678072 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 03 17:02:35.678157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:35.678130 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 03 17:02:36.747786 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:36.747737 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:02:36.750283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:36.750256 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:02:36.987811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:36.987775 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:38.920743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:38.920707 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps"] Nov 03 17:02:38.926884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:38.926859 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:38.946691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:38.946661 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps"] Nov 03 17:02:38.952588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:38.952563 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ca9d7c-serving-cert\"" Nov 03 17:02:39.030618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.030587 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.030782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.030633 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556dv\" (UniqueName: \"kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.030825 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.030805 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.030878 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.030863 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.055469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.055401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:39.132979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.132928 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.133196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.133046 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.133196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.133181 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.133362 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.133224 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556dv\" (UniqueName: \"kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.136899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.134457 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.165449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.160379 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.165449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.160945 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.173498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.170645 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556dv\" (UniqueName: \"kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv\") pod \"maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.223454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.213139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:39.241843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.241798 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:39.578601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.578571 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps"] Nov 03 17:02:39.579557 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:02:39.579532 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d620953_d68f_4c88_b01c_d31fc541dd04.slice/crio-91a8cd38f7bfb065569002898987b1b0589eff2e9bcce05ffba769ea76238ac1 WatchSource:0}: Error finding container 91a8cd38f7bfb065569002898987b1b0589eff2e9bcce05ffba769ea76238ac1: Status 404 returned error can't find the container with id 91a8cd38f7bfb065569002898987b1b0589eff2e9bcce05ffba769ea76238ac1 Nov 03 17:02:39.933111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:39.933024 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" event={"ID":"3d620953-d68f-4c88-b01c-d31fc541dd04","Type":"ContainerStarted","Data":"91a8cd38f7bfb065569002898987b1b0589eff2e9bcce05ffba769ea76238ac1"} Nov 03 17:02:40.942128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:40.942087 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" event={"ID":"3d620953-d68f-4c88-b01c-d31fc541dd04","Type":"ContainerStarted","Data":"3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955"} Nov 03 17:02:40.943137 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:40.943106 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:40.990497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:40.990450 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" podStartSLOduration=2.610105731 podStartE2EDuration="2.990435902s" totalImagesPullingTime="380.330171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:40.989284528 +0000 UTC m=+2904.682555548" watchObservedRunningTime="2026-11-03 17:02:40.990435902 +0000 UTC m=+2904.683706913" Nov 03 17:02:41.081286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:41.081251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:42.714450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:42.705340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:43.890632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:43.890598 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:44.090017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:44.089985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:44.252875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:44.252773 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:44.422211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:44.422171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:44.574003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:44.573971 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:44.749864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:44.749833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:45.429828 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:45.429778 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:45.606222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:45.606058 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:46.591269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:46.591160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:46.966197 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:46.966085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:47.964829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:47.964802 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:02:48.406382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:48.406343 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:49.100116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.100085 4539 generic.go:356] "Generic (PLEG): container finished" podID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerID="6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d" exitCode=0 Nov 03 17:02:49.100578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.100159 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" event={"ID":"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3","Type":"ContainerDied","Data":"6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d"} Nov 03 17:02:49.100708 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.100694 4539 scope.go:122] "RemoveContainer" containerID="6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d" Nov 03 17:02:49.102943 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.102914 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:49.103021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.103005 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:49.103080 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.103063 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:49.973159 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:49.973125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:50.137864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:50.137832 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" event={"ID":"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3","Type":"ContainerStarted","Data":"2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd"} Nov 03 17:02:50.891257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:50.891219 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:02:50.892767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:50.892739 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 17:02:50.892915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:50.892792 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 17:02:51.518376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:51.518284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:51.891812 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:51.891776 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 17:02:51.892007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:51.891841 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 17:02:52.892166 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:52.892127 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 17:02:52.892655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:52.892184 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 17:02:53.077288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.077238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:53.123488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.123384 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:02:53.560612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.560579 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7"] Nov 03 17:02:53.566373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.566349 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.579752 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.579724 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.579914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.579768 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.579914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.579795 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.579914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.579827 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.580107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.579964 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.580107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.580012 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.580107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.580089 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.580281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.580249 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wp9j\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.580341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.580317 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.593857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.593827 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7"] Nov 03 17:02:53.682149 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682114 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682149 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682151 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682170 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682196 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682216 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682243 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682273 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682412 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wp9j\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682491 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682526 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682842 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.682942 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.682881 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.683209 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.683128 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.683376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.683327 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.684940 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.684915 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.685391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.685372 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.720141 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.720104 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.721025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.720997 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wp9j\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.881069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.880983 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:53.891457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.891411 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 17:02:53.891583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:53.891482 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 17:02:54.244226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.244142 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7"] Nov 03 17:02:54.249004 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:02:54.248955 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1175722_921a_4307_9a59_cbbb8c6a48c9.slice/crio-2f706f7e59a2a3a68c969e6704b2ab4de858e34874d4c0842a8834224a199468 WatchSource:0}: Error finding container 2f706f7e59a2a3a68c969e6704b2ab4de858e34874d4c0842a8834224a199468: Status 404 returned error can't find the container with id 2f706f7e59a2a3a68c969e6704b2ab4de858e34874d4c0842a8834224a199468 Nov 03 17:02:54.252072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.252040 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:54.252170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.252112 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:54.252170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.252142 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:02:54.776093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.775988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:54.891910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.891871 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 17:02:54.892082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:54.891924 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 17:02:55.183184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.183072 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" event={"ID":"b1175722-921a-4307-9a59-cbbb8c6a48c9","Type":"ContainerStarted","Data":"018f86b08eaf1d832c7b72cf32447a7e27fe4d3cf5f8c0833adadf6ebb9c5414"} Nov 03 17:02:55.183184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.183139 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" event={"ID":"b1175722-921a-4307-9a59-cbbb8c6a48c9","Type":"ContainerStarted","Data":"2f706f7e59a2a3a68c969e6704b2ab4de858e34874d4c0842a8834224a199468"} Nov 03 17:02:55.243015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.242936 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podStartSLOduration=2.242913796 podStartE2EDuration="2.242913796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:55.238532347 +0000 UTC m=+2918.931803370" watchObservedRunningTime="2026-11-03 17:02:55.242913796 +0000 UTC m=+2918.936184819" Nov 03 17:02:55.881476 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.881414 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:02:55.883039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.883014 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:02:55.883207 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.883068 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:02:55.896366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.896334 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:02:55.897535 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:55.897513 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:02:56.169577 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:02:56.169498 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:02:56.473011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:56.472916 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:56.882502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:56.882456 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:02:56.883038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:56.882549 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:02:57.881675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:57.881636 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:02:57.881867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:57.881706 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:02:58.391930 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:58.391896 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:02:58.882163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:58.882133 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:02:58.882356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:58.882199 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:02:59.881727 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:59.881684 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:02:59.882139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:02:59.881745 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:03:00.670742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:00.670695 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:00.882397 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:00.882354 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:03:00.882948 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:00.882417 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:03:01.887107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:01.887074 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:03:01.887548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:01.887534 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:03:01.888260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:01.888244 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:03:02.707628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.707551 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6"] Nov 03 17:03:02.719450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.713956 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.730097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.729936 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fdbf0c-serving-cert\"" Nov 03 17:03:02.758412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.757952 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6"] Nov 03 17:03:02.792170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.792110 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.792392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.792183 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.792612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.792580 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.792793 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.792774 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72hx\" (UniqueName: \"kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.838371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.838327 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:02.894709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.894670 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72hx\" (UniqueName: \"kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.895226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.894732 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.895226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.894761 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.895226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.894913 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.895736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.895709 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.897498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.897473 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.897680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.897530 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.916773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.916733 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72hx\" (UniqueName: \"kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx\") pod \"maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:02.985553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:02.985447 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:03.062012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:03.061701 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:03.476451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:03.471613 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:03.613449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:03.607312 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6"] Nov 03 17:03:04.391473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:04.389154 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" event={"ID":"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375","Type":"ContainerStarted","Data":"5bc231e58aada3e55ff1d8fb8beac4cec51d5187568b639bd3aef7adf5992cea"} Nov 03 17:03:04.631450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:04.630046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:04.939451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:04.938818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:05.402684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:05.402640 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" event={"ID":"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375","Type":"ContainerStarted","Data":"a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a"} Nov 03 17:03:05.404403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:05.404376 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:05.445697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:05.445601 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" podStartSLOduration=2.824381664 podStartE2EDuration="3.445581047s" totalImagesPullingTime="621.199383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:05.445004857 +0000 UTC m=+2929.138275904" watchObservedRunningTime="2026-11-03 17:03:05.445581047 +0000 UTC m=+2929.138852072" Nov 03 17:03:06.792572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:06.792534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:07.035543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:07.035494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:07.226545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:07.224462 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:07.910787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:07.910662 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.078460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.078399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.250506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.250392 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.415956 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.415891 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.571532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.571501 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.596540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.596503 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz"] Nov 03 17:03:08.619191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.619161 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.648821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.648790 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz"] Nov 03 17:03:08.662301 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.662230 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.662581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.662559 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.662751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.662719 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.662849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.662829 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.662919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.662871 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.663003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.662986 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.663093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.663078 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.663157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.663106 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflvp\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.663210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.663175 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.764938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.764906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.765757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.765732 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.765972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.765883 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.765972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.765925 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.766154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.766048 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.766154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.766094 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.766154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.766144 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.766331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.766244 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.766444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.766365 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.766444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.766397 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflvp\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.767309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.767274 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.767839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.767805 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.768405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.768369 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.768694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.768676 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.768949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.768935 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.771288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.771263 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.772320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.772271 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.790130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.790090 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.790393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.790355 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflvp\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:08.936504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.934500 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:08.950388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:08.950362 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:09.391363 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:09.391326 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:09.549523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:09.549475 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:09.756440 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:09.756381 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz"] Nov 03 17:03:09.882574 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:03:09.881912 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59c0609_bbd8_4872_9a9c_071780f69931.slice/crio-d615ba5cbec3d11e30d6c1f88192ae92f2b0ed3b77d3d04d131f45dfd683f670 WatchSource:0}: Error finding container d615ba5cbec3d11e30d6c1f88192ae92f2b0ed3b77d3d04d131f45dfd683f670: Status 404 returned error can't find the container with id d615ba5cbec3d11e30d6c1f88192ae92f2b0ed3b77d3d04d131f45dfd683f670 Nov 03 17:03:09.886499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:09.885714 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:03:09.886499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:09.885811 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:03:09.886499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:09.885853 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:03:10.482130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:10.482095 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" event={"ID":"b59c0609-bbd8-4872-9a9c-071780f69931","Type":"ContainerStarted","Data":"83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8"} Nov 03 17:03:10.482642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:10.482146 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" event={"ID":"b59c0609-bbd8-4872-9a9c-071780f69931","Type":"ContainerStarted","Data":"d615ba5cbec3d11e30d6c1f88192ae92f2b0ed3b77d3d04d131f45dfd683f670"} Nov 03 17:03:10.526990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:10.526907 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podStartSLOduration=2.5268884419999997 podStartE2EDuration="2.526888442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:10.524617714 +0000 UTC m=+2934.217888737" watchObservedRunningTime="2026-11-03 17:03:10.526888442 +0000 UTC m=+2934.220159465" Nov 03 17:03:10.947306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:10.947269 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:10.951148 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:10.951122 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:11.491452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:11.490712 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:11.952840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:11.952803 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 17:03:11.953023 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:11.952863 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 17:03:12.443789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:12.443760 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:03:12.950812 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:12.950762 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 17:03:12.951301 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:12.950843 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 17:03:13.344589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:13.344464 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:13.717977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:13.717890 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:13.951450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:13.951319 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 17:03:13.951450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:13.951380 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 17:03:14.370597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:14.370561 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:03:14.370780 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:14.370628 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:03:14.951484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:14.951451 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 17:03:14.951902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:14.951505 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 17:03:15.112024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.111932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:15.634768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.634726 4539 generic.go:356] "Generic (PLEG): container finished" podID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerID="834af19cd358133b3588d89c8166ae73ad57a42726fd3e397191dd79c4a81d17" exitCode=0 Nov 03 17:03:15.634995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.634797 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" event={"ID":"f7b13c5a-b79f-40ef-98bc-b0606db9772e","Type":"ContainerDied","Data":"834af19cd358133b3588d89c8166ae73ad57a42726fd3e397191dd79c4a81d17"} Nov 03 17:03:15.635364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.635345 4539 scope.go:122] "RemoveContainer" containerID="834af19cd358133b3588d89c8166ae73ad57a42726fd3e397191dd79c4a81d17" Nov 03 17:03:15.637461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.637399 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:03:15.637552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.637491 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:03:15.637552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.637526 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:03:15.953523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.952583 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 17:03:15.953523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:15.952641 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 17:03:16.651016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:16.650981 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" event={"ID":"f7b13c5a-b79f-40ef-98bc-b0606db9772e","Type":"ContainerStarted","Data":"d4a0c8cded6ec9b05a7df192de8882b48e76e0e6226fb22aee5e2488fc2195c9"} Nov 03 17:03:16.956855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:16.956766 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:16.957324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:16.957133 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:16.958075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:16.958054 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:03:17.133941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:17.133904 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:17.209209 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:17.209117 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:03:17.214461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:17.209561 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 17:03:17.214804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:17.214768 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 17:03:18.193668 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:18.193629 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 17:03:18.194205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:18.193708 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 17:03:19.073676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:19.073643 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:19.193460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:19.193378 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 17:03:19.193686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:19.193485 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 17:03:20.193608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:20.193574 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 17:03:20.194008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:20.193655 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 17:03:20.689567 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:20.689534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:20.785186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:20.785149 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg"] Nov 03 17:03:20.785412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:20.785370 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" containerID="cri-o://cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055" gracePeriod=30 Nov 03 17:03:21.193281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:21.193246 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 17:03:21.193486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:21.193304 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 17:03:22.193947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:22.193910 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 03 17:03:22.194489 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:22.193961 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 03 17:03:22.354258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:22.354208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:23.198183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:23.198150 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:03:23.198663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:23.198462 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:03:23.199127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:23.199104 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:03:24.064540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:24.064511 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:25.674168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:25.674133 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:26.075553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.075528 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 17:03:26.133405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133365 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.133628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133503 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.133628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133570 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.133628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133602 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.133803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133688 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.133803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133724 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.133803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133741 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.133803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133778 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.134010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133865 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.134010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.133928 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token\") pod \"a3db4b84-7faf-49de-b301-c3143c8c9b88\" (UID: \"a3db4b84-7faf-49de-b301-c3143c8c9b88\") " Nov 03 17:03:26.134124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134039 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:03:26.134290 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134222 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.134388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134326 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.134725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134705 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.134832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134730 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4b84-7faf-49de-b301-c3143c8c9b88-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.134832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134743 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.134832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.134754 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.136507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.136456 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.147270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.146367 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:26.147270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.147179 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.147532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.147319 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "kube-api-access-kcfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:26.148553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.148512 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo" pod "a3db4b84-7faf-49de-b301-c3143c8c9b88" (UID: "a3db4b84-7faf-49de-b301-c3143c8c9b88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:03:26.175828 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:03:26.175790 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:03:26.238286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.238128 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.238286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.238160 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.238286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.238176 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.238286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.238190 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcfg5\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-kube-api-access-kcfg5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.238286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.238203 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db4b84-7faf-49de-b301-c3143c8c9b88-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.733207 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.733175 4539 generic.go:356] "Generic (PLEG): container finished" podID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerID="cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055" exitCode=0 Nov 03 17:03:26.733705 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.733240 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" Nov 03 17:03:26.733705 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.733257 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" event={"ID":"a3db4b84-7faf-49de-b301-c3143c8c9b88","Type":"ContainerDied","Data":"cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055"} Nov 03 17:03:26.733705 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.733324 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg" event={"ID":"a3db4b84-7faf-49de-b301-c3143c8c9b88","Type":"ContainerDied","Data":"7803efc3b1fa2012139e8ba0f603146d70b35eca0bdd61389638af5d53c10d5e"} Nov 03 17:03:26.733705 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.733355 4539 scope.go:122] "RemoveContainer" containerID="cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055" Nov 03 17:03:26.745387 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.745352 4539 scope.go:122] "RemoveContainer" containerID="cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055" Nov 03 17:03:26.745701 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:03:26.745677 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055\": container with ID starting with cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055 not found: ID does not exist" containerID="cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055" Nov 03 17:03:26.745868 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.745713 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055"} err="failed to get container status \"cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055\": rpc error: code = NotFound desc = could not find container \"cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055\": container with ID starting with cd0d48b506992110704072784659999ca92e2b2359818f3613891e2aff0fb055 not found: ID does not exist" Nov 03 17:03:26.792835 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.792804 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg"] Nov 03 17:03:26.802199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:26.802171 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sf4sg"] Nov 03 17:03:27.399693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:27.399645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:28.470882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:28.470844 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" path="/var/lib/kubelet/pods/a3db4b84-7faf-49de-b301-c3143c8c9b88/volumes" Nov 03 17:03:29.186242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:29.186192 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:30.708612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:30.708566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:32.279157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:32.279102 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:32.918581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:32.918521 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:33.073781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:33.073745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:33.227870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:33.227753 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:33.530710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:33.530674 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:33.699941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:33.699899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:34.115497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:34.115196 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:34.295032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:34.294981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:34.450101 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:34.450010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:35.794085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:35.794044 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:36.154211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:36.154085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:37.928691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:37.928654 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:39.732639 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:39.732582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:41.500634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:41.500596 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:43.295348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:43.295281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:44.994845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:44.994806 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:46.779551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:46.779515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:48.396708 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:48.396665 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:50.371598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:50.371561 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:52.039280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.039241 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp"] Nov 03 17:03:52.039980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.039966 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" Nov 03 17:03:52.040034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.039982 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" Nov 03 17:03:52.040099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.040088 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3db4b84-7faf-49de-b301-c3143c8c9b88" containerName="istio-proxy" Nov 03 17:03:52.043907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.043890 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.056669 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.056639 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-790111-serving-cert\"" Nov 03 17:03:52.076760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.076724 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp"] Nov 03 17:03:52.118872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.118830 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.119123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.118969 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.119123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.119010 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.119123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.119115 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvkp\" (UniqueName: \"kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.170295 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.170242 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:52.221845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.221794 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvkp\" (UniqueName: \"kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.222116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.222045 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.222249 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.222151 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.222249 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.222180 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.223126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.223097 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.223256 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:03:52.223217 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-serving-cert: secret "maas-api-e2e-extmod-b-790111-serving-cert" not found Nov 03 17:03:52.223324 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:03:52.223292 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls podName:c711f644-5999-424c-9af3-2f43cb718841 nodeName:}" failed. No retries permitted until 2026-11-03 17:03:52.723271687 +0000 UTC m=+2976.416542690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls") pod "maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" (UID: "c711f644-5999-424c-9af3-2f43cb718841") : secret "maas-api-e2e-extmod-b-790111-serving-cert" not found Nov 03 17:03:52.228578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.228547 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.250843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.250809 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvkp\" (UniqueName: \"kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.325210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.325114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:52.500125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.500089 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:52.728321 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.728228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:52.729206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.729182 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.731951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.731923 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\") pod \"maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:52.959820 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:52.959773 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:53.027195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.027145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:53.194225 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.194196 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:53.360687 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:03:53.360649 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc711f644_5999_424c_9af3_2f43cb718841.slice/crio-9eaf04e0b0798404a62329d2ec2edaf0fbb6c18c0525810bef78921e6bfc5280 WatchSource:0}: Error finding container 9eaf04e0b0798404a62329d2ec2edaf0fbb6c18c0525810bef78921e6bfc5280: Status 404 returned error can't find the container with id 9eaf04e0b0798404a62329d2ec2edaf0fbb6c18c0525810bef78921e6bfc5280 Nov 03 17:03:53.364060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.363922 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:53.364755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.364731 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp"] Nov 03 17:03:53.378052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.378024 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:03:53.813327 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.813290 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:53.968561 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:53.968531 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:54.041204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:54.029964 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" event={"ID":"c711f644-5999-424c-9af3-2f43cb718841","Type":"ContainerStarted","Data":"2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559"} Nov 03 17:03:54.041204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:54.030030 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" event={"ID":"c711f644-5999-424c-9af3-2f43cb718841","Type":"ContainerStarted","Data":"9eaf04e0b0798404a62329d2ec2edaf0fbb6c18c0525810bef78921e6bfc5280"} Nov 03 17:03:54.041204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:54.031243 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:03:54.231673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:54.231618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:55.448673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:55.448641 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:56.158893 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:03:56.158860 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:03:56.874320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:56.874268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:03:58.573286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:03:58.573250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:00.184723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:00.184682 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:01.063331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:01.063302 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:04:01.110251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:01.110192 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" podStartSLOduration=8.746804176 podStartE2EDuration="9.11017724s" totalImagesPullingTime="363.373064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:54.103109282 +0000 UTC m=+2977.796380306" watchObservedRunningTime="2026-11-03 17:04:01.11017724 +0000 UTC m=+2984.803448295" Nov 03 17:04:01.790107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:01.790038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:03.369721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:03.369677 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:04.906164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:04.906125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:06.444761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:06.444728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:08.098759 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:08.098727 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:09.782473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:09.782356 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:11.448841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:11.448805 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:13.049574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:13.049534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:13.833442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:13.833383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:14.743360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:14.743323 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:16.666442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:16.666383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:17.058145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:17.058113 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:04:17.072894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:17.072865 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:04:18.237778 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:18.237745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:18.387812 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:18.387723 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:18.577450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:18.577278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:18.733023 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:18.732900 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:19.395979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:19.395946 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:19.557208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:19.557174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:20.379292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:20.379264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:20.531836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:20.531790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:22.208751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:22.208704 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:23.785561 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:23.785524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:25.475779 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:25.475713 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:26.160859 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:04:26.160820 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:04:27.110003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:27.108849 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:28.709027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:28.708821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:30.474136 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:30.474068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:32.062966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:32.062929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:33.682712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:33.682680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:35.415478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:35.415393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:37.326256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:37.326218 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:39.079286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:39.079246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:39.234724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:39.234690 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:39.387026 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:39.386926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:40.973792 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:40.973754 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:41.127300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:41.127260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:41.762712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:41.762674 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:43.291253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:43.291213 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:44.948169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:44.948132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:45.641021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:45.640986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:45.829184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:45.829137 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:46.106483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:46.106450 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:46.285061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:46.285027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:46.468683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:46.468588 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:46.916634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:46.916594 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:47.085411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:47.085378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:48.682634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:48.682599 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:50.259543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:50.259511 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:51.927751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:51.927717 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:53.450985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:53.450950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:55.088512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:55.088416 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:56.160669 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:04:56.160631 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:04:56.704055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:56.704002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:58.301139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:58.301098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:59.271647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:59.271604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:59.426608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:59.426538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:04:59.905777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:04:59.905733 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:00.075771 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:00.075684 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:01.693763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:01.693722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:03.180093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:03.180051 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:04.866020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:04.865987 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:06.378415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:06.378383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:06.533270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:06.533234 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:06.685793 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:06.685706 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:07.997300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:07.997268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:10.082627 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:10.082592 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:10.225077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:10.225035 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:10.405481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:10.405358 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:12.032692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:12.032643 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr"] Nov 03 17:05:12.042733 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:12.042699 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411f3e66-7w5hr"] Nov 03 17:05:12.177712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:12.177655 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:12.470588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:12.470495 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27168524-3ef5-4c01-aa8b-958ba9b4e0c7" path="/var/lib/kubelet/pods/27168524-3ef5-4c01-aa8b-958ba9b4e0c7/volumes" Nov 03 17:05:12.668611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:12.668574 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:12.978975 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:12.978938 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:13.606824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:13.606787 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:13.753276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:13.753239 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:13.915281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:13.915187 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:14.077390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:14.077356 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:14.809831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:14.809734 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:15.540298 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:15.540271 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:15.722884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:15.722848 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:16.524928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:16.524886 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:16.670345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:16.670313 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:17.118364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:17.118327 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:17.388581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:17.388483 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:19.083570 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:19.083495 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:20.169841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:20.169808 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:20.335259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:20.335226 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:21.317187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:21.317152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:21.472496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:21.472393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:21.867797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:21.867759 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:22.039874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:22.039825 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:23.824199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:23.824166 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:24.027606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:24.027572 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:24.199831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:24.199540 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:24.354541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:24.354503 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:24.781366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:24.781334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:24.945351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:24.945305 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:25.117504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:25.115025 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:25.938496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:25.938385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:26.122923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:26.122877 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:26.163665 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:05:26.163588 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:05:26.277576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:26.277545 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:28.177587 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:28.177553 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:28.941045 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:28.941001 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:29.094194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:29.094159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:29.787382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:29.787353 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:31.385006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:31.384969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:32.270627 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:32.270578 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:32.423183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:32.423150 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:32.892046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:32.892009 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:33.281827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:33.281000 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:35.135012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:35.134975 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:36.992817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:36.992736 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:38.582258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:38.582219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:38.736680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:38.736646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:38.886081 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:38.885996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:39.034590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.034545 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:39.350331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.350295 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm"] Nov 03 17:05:39.362852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.362805 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.376127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.376098 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 17:05:39.376291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.376131 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 17:05:39.376291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.376161 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 17:05:39.376396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.376291 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xzr7j\"" Nov 03 17:05:39.391842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.391814 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm"] Nov 03 17:05:39.520595 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.520554 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.520806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.520673 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.520806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.520723 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.520806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.520764 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.520806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.520783 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.521027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.520962 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.521115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.521097 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqkp\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.622465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622329 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.622465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622465 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.623057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622511 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.623057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622558 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.623057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622584 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.623057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622631 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.623057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.622753 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqkp\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.623367 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.623276 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.625274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.625244 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.625576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.625554 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.625793 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.625776 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.626089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.626068 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.644747 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.644714 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.644978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.644952 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqkp\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfwwm\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:39.677081 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:39.677042 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:40.006190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.006154 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm"] Nov 03 17:05:40.007621 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:05:40.007584 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1329b928_6285_4d41_8a3e_af0b55343c16.slice/crio-9cd588944c7b26ee64cb8535ae9868aef5191a6b18aaed9e3e54b3d358934300 WatchSource:0}: Error finding container 9cd588944c7b26ee64cb8535ae9868aef5191a6b18aaed9e3e54b3d358934300: Status 404 returned error can't find the container with id 9cd588944c7b26ee64cb8535ae9868aef5191a6b18aaed9e3e54b3d358934300 Nov 03 17:05:40.010611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.010576 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:05:40.010713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.010698 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:05:40.152448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.149889 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" event={"ID":"1329b928-6285-4d41-8a3e-af0b55343c16","Type":"ContainerStarted","Data":"174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842"} Nov 03 17:05:40.152448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.149957 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" event={"ID":"1329b928-6285-4d41-8a3e-af0b55343c16","Type":"ContainerStarted","Data":"9cd588944c7b26ee64cb8535ae9868aef5191a6b18aaed9e3e54b3d358934300"} Nov 03 17:05:40.152448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.150456 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:40.163229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.163198 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:40.209501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:40.201754 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" podStartSLOduration=1.201731467 podStartE2EDuration="1.201731467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:40.192622765 +0000 UTC m=+3083.885893801" watchObservedRunningTime="2026-11-03 17:05:40.201731467 +0000 UTC m=+3083.895002490" Nov 03 17:05:41.163278 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:41.163246 4539 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rfwwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:05:41.163839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:41.163307 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" podUID="1329b928-6285-4d41-8a3e-af0b55343c16" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:05:41.799697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:41.798767 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:43.481294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:43.481263 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:44.162624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:44.162590 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:05:45.132497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:45.132456 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:45.296230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:45.296190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:45.611507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:45.611472 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:45.823585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:45.823447 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:46.856783 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:46.856747 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:47.339347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:47.339314 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:49.171260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:49.171230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:50.773709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:50.773671 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:51.271877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:51.271777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:52.485774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:52.485742 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:54.083197 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:54.083160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:55.688121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:55.688083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:56.158962 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:05:56.158927 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:05:57.275144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:57.275098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:05:58.847274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:05:58.847159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:00.497196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:00.497158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:02.146780 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:02.146728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:03.700406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:03.700333 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:05.288389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:05.288357 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:06.835284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:06.835195 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:08.518791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:08.518757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:10.866306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:10.866271 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:12.469927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:12.469897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:12.616088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:12.616053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:14.063269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:14.063224 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:14.253686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:14.253644 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:15.888265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:15.888216 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:17.505195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:17.505145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:18.106397 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:18.106359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:18.107845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:18.107824 4539 scope.go:122] "RemoveContainer" containerID="0ae333b52e15424a3bf667f5c7d84fbc886e23713a318fefe2aa4a6a2f5aa725" Nov 03 17:06:18.249589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:18.249553 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:19.123606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:19.123574 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:19.764821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:19.764790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:21.394081 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:21.394049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:23.014689 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:23.014654 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:24.592011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:24.591973 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:25.029778 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:25.029743 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:25.173257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:25.173224 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:25.333116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:25.333024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:26.160321 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:06:26.160280 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:06:26.261099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:26.261073 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:27.897365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:27.897329 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:29.491950 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:29.491898 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:30.617854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:30.617815 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:30.772020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:30.771949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:31.499261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:31.499206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:32.361606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:32.361572 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:33.362147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:33.362109 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:34.970858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:34.970821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:36.562441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:36.562379 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:38.145654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:38.145616 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:38.553184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:38.553149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:38.701480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:38.701446 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:39.777297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:39.777258 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:41.379762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:41.379719 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:42.953257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:42.953228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:44.678414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:44.678380 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:46.195499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:46.195457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:47.725503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:47.725455 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:49.395010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:49.394979 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:50.998147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:50.998107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:52.600384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:52.600342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:54.187016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:54.186981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:55.819224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:55.813656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:56.159300 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:06:56.159212 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:06:57.326299 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:57.326259 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:06:58.901195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:06:58.901096 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:00.499640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:00.499516 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:02.042154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:02.042110 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:03.635498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:03.635463 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:05.520113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:05.520075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:07.455769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:07.455676 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:08.731656 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:08.731612 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:08.887060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:08.887026 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:09.045892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:09.045856 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:09.396607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:09.396518 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:10.737817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:10.737781 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:10.890543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:10.890496 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:11.654210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:11.654179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:13.095663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:13.095628 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:13.243690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:13.243649 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:13.466950 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:13.466859 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:13.750954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:13.750866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:14.194794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:14.194762 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:15.285005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:15.284933 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:16.944324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:16.944288 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:18.411980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:18.411947 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:18.560196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:18.560142 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:18.837205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:18.837066 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:18.992996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:18.992954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:19.323361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:19.323319 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:19.615191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:19.615077 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:19.795539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:19.795506 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:19.957846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:19.957740 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:20.134193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:20.134158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:20.755960 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:20.755918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:20.901085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:20.901034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:21.542339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:21.542293 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:21.719951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:21.719918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:21.872053 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:21.871894 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:22.359835 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:22.359801 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:23.431079 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:23.431041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:25.377541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:25.377404 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:26.158976 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:07:26.158938 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:07:26.981057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:26.981010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:28.411363 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:28.411328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:28.573255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:28.573181 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:28.763512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:28.753895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:28.912301 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:28.912268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:29.097011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:29.096975 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:29.265142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:29.265107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:29.442309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:29.442221 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:30.218051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:30.217968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:30.373015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:30.372984 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:31.164239 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:31.164206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:31.327449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:31.325897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:31.499525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:31.499414 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:31.737562 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:31.737528 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:31.994188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:31.994155 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:32.191881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:32.191842 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:32.389907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:32.389869 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:32.564244 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:32.564203 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:32.714594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:32.714508 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:32.862640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:32.862602 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:33.024337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:33.024300 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:33.290004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:33.289908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:34.019129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:34.019084 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:34.175778 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:34.175741 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:34.348449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:34.338562 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:35.205215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:35.205184 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:36.270407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:36.270361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:37.945176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:37.945141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:39.409620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:39.409582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:39.595457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:39.594996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:39.770085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:39.769994 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:41.680536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:41.680504 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:43.277450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:43.272657 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:45.019448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:45.013512 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:46.596533 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:46.596504 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:48.282611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:48.282573 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:49.898236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:49.895061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:51.497095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:51.497064 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:53.170158 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:53.170117 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:54.788309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:54.788271 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:56.158702 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:07:56.158659 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:07:56.337711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:56.337656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:57.995526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:57.995486 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:59.521098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:59.521062 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:07:59.877102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:07:59.877068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:01.055442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:01.055390 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:02.644728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:02.644689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:04.501834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:04.501801 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:06.159881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:06.159836 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:06.751757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:06.751714 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:07.914969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:07.914932 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:08.085450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:08.084615 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:09.853344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:09.853307 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:11.354731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:11.354691 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:12.596177 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:12.596130 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:12.884345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:12.884260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:14.318736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:14.318702 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:14.464101 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:14.464072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:16.070599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:16.070563 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:17.612409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:17.612375 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:18.144234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:18.144188 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:18.296315 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:18.296286 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:19.263996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:19.263964 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:19.418134 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:19.418103 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:20.990231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:20.990192 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:22.583851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:22.583803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:24.278435 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:24.278385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:25.284495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:25.284399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:25.596260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:25.596168 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:25.964381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:25.958312 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:26.159277 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:08:26.159235 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:08:27.554311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:27.554280 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:29.195783 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:29.195745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:30.453450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:30.452983 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:30.617607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:30.617569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:31.010210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:31.010176 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:32.091324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:32.091276 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:32.849968 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:32.849934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:34.499237 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:34.499201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:36.105451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:36.100520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:37.720257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:37.720188 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:37.904094 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:37.904024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:39.328092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:39.328050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:40.958149 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:40.958107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:42.623321 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:42.618061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:44.377832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:44.377793 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:46.015588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:46.015554 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:47.639635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:47.639600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:49.022762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:49.022731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:49.215048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:49.214998 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:50.755631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:50.755596 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:51.123834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:51.123796 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:51.263618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:51.263582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:52.454623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:52.454573 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:53.898711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:53.898678 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:54.435704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:54.435669 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:54.581462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:54.581405 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:55.624965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:55.624937 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:56.020394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:56.020312 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:56.305184 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:08:56.305147 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:08:57.879921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:57.879885 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:59.586849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:59.586775 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:59.804851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:59.803964 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:08:59.946336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:08:59.946245 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:01.580937 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:01.580895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:01.932504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:01.932364 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:02.089951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:02.089921 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:03.317664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:03.316599 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:04.998220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:04.998185 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:06.681380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:06.681346 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:08.183602 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:08.183563 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:09.698192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:09.698139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:11.320495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:11.312244 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:12.765475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:12.765391 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:12.913078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:12.912978 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:13.146463 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:13.141869 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:13.484967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:13.484453 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:13.780497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:13.776545 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:14.048524 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:14.048372 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:14.190966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:14.190915 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:14.460027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:14.459936 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:14.631121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:14.631087 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:15.456399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:15.456368 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:15.609379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:15.609342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:15.916980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:15.916945 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:16.100350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:16.100316 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:16.434829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:16.434787 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:17.115547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:17.115517 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:09:17.132711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:17.132685 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:09:17.767448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:17.767109 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:19.451398 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:19.451353 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:21.202367 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:21.202328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:22.529402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:22.529365 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:22.670258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:22.670226 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:22.890529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:22.890165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:24.394833 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:24.394799 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:25.274095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:25.274056 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:25.442502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:25.430916 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:26.179137 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:09:26.179098 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:09:26.475072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:26.474981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:26.623171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:26.622919 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:27.118330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:27.118295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:27.349765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:27.349707 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:28.304736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:28.304694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:29.170190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:29.170149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:29.460144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:29.460045 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:30.151181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:30.151141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:30.373889 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:30.373855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:31.864640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:31.864581 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:32.062021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:32.061975 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:33.672358 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:33.672317 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:34.237649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:34.237616 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:34.398326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:34.398288 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:35.282563 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:35.282530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:36.979657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:36.979618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:37.132032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:37.131987 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:37.281125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:37.281089 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:38.805939 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:38.805900 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:40.422454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:40.421509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:40.800861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:40.800826 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:40.960456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:40.960408 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:41.122144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:41.122034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:41.285059 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:41.284986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:41.448722 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:41.448618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:41.624378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:41.624337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:42.202723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:42.202684 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:42.359850 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:42.359813 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:43.559838 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:43.559808 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:43.712005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:43.711964 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:43.862677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:43.862586 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:44.125509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:44.124699 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:45.799485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:45.799448 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:46.073013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:46.072916 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:47.605073 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:47.605032 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:49.026495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:49.026452 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:49.179502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:49.179465 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:49.352282 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:49.351710 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:49.663657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:49.663566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:51.195681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:51.195632 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:52.727732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:52.727695 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:54.483861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:54.483030 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:56.160489 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:09:56.160343 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:09:56.354529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:56.354496 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:56.505050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:56.504951 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:56.669546 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:56.669395 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:56.952491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:56.952453 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:09:58.945811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:09:58.945753 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:00.615069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:00.615024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:02.344383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:02.344275 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:03.198170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:03.198128 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:03.361387 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:03.361353 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:03.512741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:03.512662 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:03.689453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:03.689358 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:03.843201 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:03.843166 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:04.110526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:04.110402 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:04.301998 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:04.301962 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:04.795448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:04.795391 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:04.963370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:04.963335 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:06.196448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:06.194040 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:06.351642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:06.351594 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:07.879690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:07.879654 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:09.456689 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:09.456662 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:11.161525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:11.161410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:12.904033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:12.903940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:13.189303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:13.189225 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:13.334910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:13.334884 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:14.564078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:14.564049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:14.716095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:14.716043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:14.866405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:14.866322 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:15.537028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:15.536992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:15.713455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:15.710239 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:15.946509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:15.903496 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675"] Nov 03 17:10:16.047449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.042931 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.152450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.123003 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.168693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.168662 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.168948 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.168928 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.173531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.169072 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.173842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.173816 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.175816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.152753 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675"] Nov 03 17:10:16.179578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.175831 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.179762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.179673 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.179762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.179730 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvt8t\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.179913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.179786 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.280807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.280763 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvt8t\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.280824 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.280935 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.280990 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281026 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281071 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281122 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281154 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281234 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281313 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281582 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281702 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.281782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.281722 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.282095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.282068 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.283695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.283672 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.283965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.283945 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.305384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.305356 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.307006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.306975 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvt8t\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rl675\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.488974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.488942 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:16.595413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.595356 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:16.847909 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:10:16.847882 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c7d2f5_ee54_4e58_9f7d_48987054010f.slice/crio-4b3d4754044e916f0819e4ba082decf5b1a49114ef27ff6c2887e68c3b98af39 WatchSource:0}: Error finding container 4b3d4754044e916f0819e4ba082decf5b1a49114ef27ff6c2887e68c3b98af39: Status 404 returned error can't find the container with id 4b3d4754044e916f0819e4ba082decf5b1a49114ef27ff6c2887e68c3b98af39 Nov 03 17:10:16.848245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.848025 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675"] Nov 03 17:10:16.851018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.850985 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:16.851118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.851053 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:16.851118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:16.851105 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:17.167551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:17.167409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:17.343496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:17.335248 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" event={"ID":"56c7d2f5-ee54-4e58-9f7d-48987054010f","Type":"ContainerStarted","Data":"ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241"} Nov 03 17:10:17.343789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:17.343761 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" event={"ID":"56c7d2f5-ee54-4e58-9f7d-48987054010f","Type":"ContainerStarted","Data":"4b3d4754044e916f0819e4ba082decf5b1a49114ef27ff6c2887e68c3b98af39"} Nov 03 17:10:17.489832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:17.489744 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:17.490906 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:17.490882 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:17.491007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:17.490939 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:18.489411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:18.489379 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:18.489908 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:18.489464 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:19.317450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:19.310923 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:19.489883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:19.489848 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:19.490291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:19.489895 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:20.489649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:20.489617 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:20.489829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:20.489666 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:21.083776 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:21.080096 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:21.489463 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:21.489373 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:21.489689 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:21.489666 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:21.655828 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:21.655749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:21.832985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:21.832893 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:22.489924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:22.489886 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:22.490324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:22.489933 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:22.758545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:22.753950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:22.922449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:22.911360 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:23.489452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:23.489387 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:23.489642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:23.489460 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:23.717880 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:23.717833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:23.875887 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:23.875854 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:24.489729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:24.489693 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:24.489910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:24.489751 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:24.961349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:24.961316 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:25.121660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:25.121182 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:25.282137 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:25.282109 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:25.489350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:25.489314 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:10:25.489565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:25.489363 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 17:10:25.634840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:25.634751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:25.791898 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:25.791864 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:26.164311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:26.164274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:26.166752 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:10:26.166718 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:10:27.165879 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:27.165852 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:27.166528 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:27.166302 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:27.167614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:27.167593 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:10:27.237410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:27.237349 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podStartSLOduration=12.237330252 podStartE2EDuration="12.237330252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:17.405267991 +0000 UTC m=+3361.098539014" watchObservedRunningTime="2026-11-03 17:10:27.237330252 +0000 UTC m=+3370.930601278" Nov 03 17:10:27.941382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:27.941324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:28.765632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:28.753330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:28.911985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:28.911705 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:29.460450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:29.456901 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:29.704326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:29.704292 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:30.193335 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:30.193303 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:30.746285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:30.746249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:31.026236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.026204 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:31.032551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.032479 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn"] Nov 03 17:10:31.087635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.087601 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn"] Nov 03 17:10:31.088378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.088007 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.267442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.267387 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.268085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.268058 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdh7\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.284624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.284531 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.284905 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.284885 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.285179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.285161 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.285307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.285290 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.285413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.285397 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.285724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.285700 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.286013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.285984 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.332452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.323930 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:31.399485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.399442 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.399753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.399734 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.399860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.399847 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.400046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.400034 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.400172 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.400158 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.400288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.400274 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.400461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.400443 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdh7\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.400629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.400614 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.400768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.400755 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.402297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.402273 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.414006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.402711 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.414222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.403339 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.414337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.413738 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.424862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.424821 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.437870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.437832 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.451850 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.451811 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.456242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.456138 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.477721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.477679 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdh7\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-tbxvn\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:31.522839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.522810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:31.724084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:31.723982 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:32.023515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.023401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:32.078807 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:10:32.078770 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11212458_38d0_48bd_a512_ecb6b9b24dc1.slice/crio-3d15a7c76a33af0e3bf07f8b0d38c04504faa6fcc52067ef703645082d80377e WatchSource:0}: Error finding container 3d15a7c76a33af0e3bf07f8b0d38c04504faa6fcc52067ef703645082d80377e: Status 404 returned error can't find the container with id 3d15a7c76a33af0e3bf07f8b0d38c04504faa6fcc52067ef703645082d80377e Nov 03 17:10:32.082912 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.082867 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:32.083029 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.082978 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:32.083071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.083030 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:32.085414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.085387 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn"] Nov 03 17:10:32.194447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.188079 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:32.669020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.668924 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" event={"ID":"11212458-38d0-48bd-a512-ecb6b9b24dc1","Type":"ContainerStarted","Data":"f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c"} Nov 03 17:10:32.669657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.669631 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" event={"ID":"11212458-38d0-48bd-a512-ecb6b9b24dc1","Type":"ContainerStarted","Data":"3d15a7c76a33af0e3bf07f8b0d38c04504faa6fcc52067ef703645082d80377e"} Nov 03 17:10:32.729448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.728759 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:32.757275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.757219 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podStartSLOduration=2.757203422 podStartE2EDuration="2.757203422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:32.745810241 +0000 UTC m=+3376.439081264" watchObservedRunningTime="2026-11-03 17:10:32.757203422 +0000 UTC m=+3376.450474443" Nov 03 17:10:32.871515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:32.871387 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:33.313457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:33.306580 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:33.737501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:33.736700 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:33.737501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:33.736767 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:33.758966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:33.758903 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:33.915384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:33.915320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:34.729982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:34.729947 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:34.730291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:34.730264 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:35.057250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:35.030973 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:35.739448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:35.736187 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:35.739448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:35.736265 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:36.736452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:36.725807 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:36.736452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:36.725866 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:36.832452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:36.825862 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:36.985288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:36.985250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:37.180452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:37.179490 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:37.728506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:37.728467 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:37.728810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:37.728776 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:38.677695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:38.677472 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z"] Nov 03 17:10:38.679652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:38.679623 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" containerID="cri-o://e380d1ca6a06d5c634b21cd6ae86517f88986b2b92ff1dd3f35a8f5cc7facf2f" gracePeriod=30 Nov 03 17:10:38.748000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:38.743540 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:38.748000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:38.743604 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:39.348565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:39.348524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:39.732996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:39.732887 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:39.732996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:39.732971 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:40.017810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:40.017721 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc"] Nov 03 17:10:40.018034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:40.017996 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" containerID="cri-o://ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0" gracePeriod=30 Nov 03 17:10:40.724853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:40.724808 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:40.725055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:40.724881 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:41.095375 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.095329 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:41.278989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.278948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:41.438848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.438739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:41.600665 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.600630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:41.725034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.724922 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 03 17:10:41.725034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.725007 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 03 17:10:41.776194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:41.776153 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:42.204375 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:42.204342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:42.775946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:42.775757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:42.890571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:42.890484 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:42.891796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:42.891766 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:42.893258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:42.893232 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:10:43.072586 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:43.054993 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:43.218810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:43.218775 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:43.783026 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:43.782989 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:43.964843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:43.964725 4539 generic.go:356] "Generic (PLEG): container finished" podID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerID="e380d1ca6a06d5c634b21cd6ae86517f88986b2b92ff1dd3f35a8f5cc7facf2f" exitCode=0 Nov 03 17:10:43.965568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:43.965478 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" event={"ID":"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc","Type":"ContainerDied","Data":"e380d1ca6a06d5c634b21cd6ae86517f88986b2b92ff1dd3f35a8f5cc7facf2f"} Nov 03 17:10:44.307123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.307083 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:10:44.438116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438072 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438198 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438236 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438279 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438314 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438402 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438478 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438534 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.438657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.438630 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy\") pod \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\" (UID: \"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc\") " Nov 03 17:10:44.449578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.440023 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:44.449578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.440310 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:44.449578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.440608 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:44.449578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.440671 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:44.449578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.440957 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:44.457259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.455067 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:44.457259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.457039 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "kube-api-access-xq5br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:44.461533 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.459576 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:10:44.481675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.481627 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token" pod "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" (UID: "5bb17265-a6f5-449b-b8b9-c10f2d8aadcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541544 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541581 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541598 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541612 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541628 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541642 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541657 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541671 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.543192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.541684 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5br\" (UniqueName: \"kubernetes.io/projected/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc-kube-api-access-xq5br\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:44.589006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.588921 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:44.752209 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.752169 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:44.950525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.950375 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:44.977478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.977435 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" Nov 03 17:10:44.979650 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.977412 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z" event={"ID":"5bb17265-a6f5-449b-b8b9-c10f2d8aadcc","Type":"ContainerDied","Data":"5910112388a6e7daed3e1b91d86673222f150a710bd858236f75a873711e6021"} Nov 03 17:10:44.979650 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:44.979506 4539 scope.go:122] "RemoveContainer" containerID="e380d1ca6a06d5c634b21cd6ae86517f88986b2b92ff1dd3f35a8f5cc7facf2f" Nov 03 17:10:45.048889 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.046646 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z"] Nov 03 17:10:45.056707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.056671 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-br42z"] Nov 03 17:10:45.112400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.112349 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:45.421920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.360545 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:45.590052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.571163 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:45.695522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.687111 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719244 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719294 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719347 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719499 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719651 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719715 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719810 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719903 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.720447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.719948 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket\") pod \"74e9ff44-d087-40e5-b105-07b8c01b0686\" (UID: \"74e9ff44-d087-40e5-b105-07b8c01b0686\") " Nov 03 17:10:45.725017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.722287 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:45.725017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.724530 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:45.725017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.724747 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:45.728554 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.728520 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:45.753648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.731565 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:45.771089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.771029 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:10:45.771497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.771469 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "kube-api-access-6v6hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:45.785685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.785521 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:10:45.785685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.785651 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token" pod "74e9ff44-d087-40e5-b105-07b8c01b0686" (UID: "74e9ff44-d087-40e5-b105-07b8c01b0686"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:45.822827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822793 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6hd\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-kube-api-access-6v6hd\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.822827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822827 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74e9ff44-d087-40e5-b105-07b8c01b0686-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822840 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822852 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822864 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822875 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822886 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822896 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e9ff44-d087-40e5-b105-07b8c01b0686-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.823116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.822906 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74e9ff44-d087-40e5-b105-07b8c01b0686-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:10:45.998875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.998782 4539 generic.go:356] "Generic (PLEG): container finished" podID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerID="ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0" exitCode=0 Nov 03 17:10:45.998875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.998853 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" event={"ID":"74e9ff44-d087-40e5-b105-07b8c01b0686","Type":"ContainerDied","Data":"ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0"} Nov 03 17:10:45.999088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.998904 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" event={"ID":"74e9ff44-d087-40e5-b105-07b8c01b0686","Type":"ContainerDied","Data":"900c9e5106e6ed2efa7690611702c3704f04eb1b0cd84b9709af900c4dd45242"} Nov 03 17:10:45.999088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.998932 4539 scope.go:122] "RemoveContainer" containerID="ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0" Nov 03 17:10:45.999088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:45.999064 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" Nov 03 17:10:46.061746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.057870 4539 scope.go:122] "RemoveContainer" containerID="b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa" Nov 03 17:10:46.076323 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.076299 4539 scope.go:122] "RemoveContainer" containerID="ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0" Nov 03 17:10:46.076768 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:10:46.076730 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0\": container with ID starting with ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0 not found: ID does not exist" containerID="ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0" Nov 03 17:10:46.076903 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.076780 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0"} err="failed to get container status \"ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0\": rpc error: code = NotFound desc = could not find container \"ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0\": container with ID starting with ed2131d95162f06735160a1d6c685e69c656267583bf1abc6f5bca824008a8e0 not found: ID does not exist" Nov 03 17:10:46.076903 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.076806 4539 scope.go:122] "RemoveContainer" containerID="b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa" Nov 03 17:10:46.077165 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:10:46.077137 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa\": container with ID starting with b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa not found: ID does not exist" containerID="b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa" Nov 03 17:10:46.077289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.077165 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa"} err="failed to get container status \"b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa\": rpc error: code = NotFound desc = could not find container \"b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa\": container with ID starting with b4908ea42c25cc9f129d59273ab13b68bce6dc29226f7fa4144687376c4dfbfa not found: ID does not exist" Nov 03 17:10:46.078685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.078663 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc"] Nov 03 17:10:46.083346 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.083318 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc"] Nov 03 17:10:46.086902 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:10:46.086873 4539 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e9ff44_d087_40e5_b105_07b8c01b0686.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e9ff44_d087_40e5_b105_07b8c01b0686.slice/crio-900c9e5106e6ed2efa7690611702c3704f04eb1b0cd84b9709af900c4dd45242\": RecentStats: unable to find data in memory cache]" Nov 03 17:10:46.120390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.120286 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:46.276824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.276792 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:46.476727 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.476690 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" path="/var/lib/kubelet/pods/5bb17265-a6f5-449b-b8b9-c10f2d8aadcc/volumes" Nov 03 17:10:46.477177 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.477162 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" path="/var/lib/kubelet/pods/74e9ff44-d087-40e5-b105-07b8c01b0686/volumes" Nov 03 17:10:46.511336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.511242 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:10:46.511616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:46.511360 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-nwtbc" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:47.505103 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:47.503765 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:48.390728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:48.390692 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:51.040185 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.040151 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:51.183446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.183400 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:51.828487 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.828451 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6"] Nov 03 17:10:51.829196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829176 4539 cpu_manager.go:414] "containerMap: removing container" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" Nov 03 17:10:51.829196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829195 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" Nov 03 17:10:51.829330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829214 4539 cpu_manager.go:414] "containerMap: removing container" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" Nov 03 17:10:51.829330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829219 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" Nov 03 17:10:51.829330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829251 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" Nov 03 17:10:51.829330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829256 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" Nov 03 17:10:51.829482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829342 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb17265-a6f5-449b-b8b9-c10f2d8aadcc" containerName="istio-proxy" Nov 03 17:10:51.829482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829352 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" Nov 03 17:10:51.829482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.829360 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e9ff44-d087-40e5-b105-07b8c01b0686" containerName="istio-proxy" Nov 03 17:10:51.833119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.833099 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.846723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.846696 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3a0e27-openshift-default-dockercfg-s7k4t\"" Nov 03 17:10:51.863485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.863452 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6"] Nov 03 17:10:51.900607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.900563 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.900840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.900792 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l5z\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.900915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.900864 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.901020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.900997 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.901116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.901095 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.901289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.901124 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.901289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.901153 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.901289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.901236 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:51.901529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:51.901296 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003042 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003210 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l5z\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003346 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003284 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003369 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003463 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003491 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003513 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003544 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.003652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003615 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.004388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.003953 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.004388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.004065 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.004388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.004162 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.004388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.004327 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.004753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.004622 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.006121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.006092 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.006345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.006325 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.025795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.025757 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.026050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.026027 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l5z\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.078938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.078817 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:52.148111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.147735 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:52.513923 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:10:52.513882 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode398ba67_40c5_4c34_9cc0_5c5788be6e0f.slice/crio-1b2525d7a222e83b8ee00bf80671f16bf7f4cdb83317e05de4c95d7ece52b8ce WatchSource:0}: Error finding container 1b2525d7a222e83b8ee00bf80671f16bf7f4cdb83317e05de4c95d7ece52b8ce: Status 404 returned error can't find the container with id 1b2525d7a222e83b8ee00bf80671f16bf7f4cdb83317e05de4c95d7ece52b8ce Nov 03 17:10:52.517412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.517364 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:52.517692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.517486 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:52.517692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.517511 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6"] Nov 03 17:10:52.517692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:52.517538 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:10:53.294651 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:53.294607 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" event={"ID":"e398ba67-40c5-4c34-9cc0-5c5788be6e0f","Type":"ContainerStarted","Data":"cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7"} Nov 03 17:10:53.295102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:53.294681 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" event={"ID":"e398ba67-40c5-4c34-9cc0-5c5788be6e0f","Type":"ContainerStarted","Data":"1b2525d7a222e83b8ee00bf80671f16bf7f4cdb83317e05de4c95d7ece52b8ce"} Nov 03 17:10:53.377832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:53.377763 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podStartSLOduration=2.377747003 podStartE2EDuration="2.377747003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:53.36622326 +0000 UTC m=+3397.059494282" watchObservedRunningTime="2026-11-03 17:10:53.377747003 +0000 UTC m=+3397.071018024" Nov 03 17:10:53.884679 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:53.880400 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:54.149050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:54.148955 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:54.151143 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:54.151113 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 17:10:54.151292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:54.151176 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 17:10:55.149380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:55.149338 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 17:10:55.149910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:55.149400 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 17:10:55.540449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:55.531883 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:56.149353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:56.149306 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 17:10:56.149585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:56.149379 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 17:10:56.166283 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:10:56.166238 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:10:57.149150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:57.149099 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 17:10:57.149345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:57.149186 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 17:10:57.285273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:57.285229 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:58.149008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:58.148966 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 17:10:58.149356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:58.149050 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 17:10:59.122625 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:59.122585 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:10:59.160190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:59.160159 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:59.161163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:59.161126 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:10:59.161782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:10:59.161762 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:11:00.809157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:00.809125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:02.753844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:02.753799 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:04.383533 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:04.383498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:06.455569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:06.455532 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:08.323751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:08.323712 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:10.115290 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:10.115255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:11.749246 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:11.749198 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:13.323481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:13.306985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:14.748285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:14.748241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:14.969262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:14.969229 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:15.330694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:15.330661 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:16.518559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:16.518522 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:18.280333 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:18.277687 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:19.909370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:19.901020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:21.465760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:21.465702 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:23.056989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:23.056952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:24.578663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:24.578630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:26.158364 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:11:26.158322 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:11:26.166719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:26.166689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:27.723459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:27.720153 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:29.313828 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:29.313793 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:30.907447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:30.906055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:32.566128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:32.566090 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:34.160700 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:34.160668 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:35.803968 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:35.795973 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:37.497765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:37.497731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:39.138147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:39.138108 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:40.576985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:40.576936 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:42.183895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:42.183857 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:43.835745 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:43.835711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:45.495909 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:45.484056 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:47.090102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:47.090038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:48.690097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:48.690056 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:50.306964 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:50.306848 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:51.845001 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:51.844959 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:53.498832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:53.498792 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:55.101608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:55.101580 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:56.163836 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:11:56.163800 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:11:56.683241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:56.683205 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:11:58.373374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:11:58.373328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:00.093236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:00.086442 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:01.906683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:01.904007 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:02.061361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:02.061328 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:02.230501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:02.229934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:03.187686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:03.187650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:04.092345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:04.092305 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:05.818750 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:05.818717 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:07.436241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:07.436210 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:08.968026 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:08.967992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:10.497504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:10.497470 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:12.182667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:12.182638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:13.735588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:13.735533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:14.801281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:14.801244 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:15.311849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:15.311808 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:16.969526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:16.969478 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:17.655684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:17.655646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:17.804941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:17.804908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:18.258041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:18.258007 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:18.696950 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:18.696915 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:20.318402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:20.318367 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:21.821927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:21.821881 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:23.358216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:23.358186 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:24.805846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:24.805809 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:24.967995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:24.967964 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:25.115768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:25.115675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:26.165862 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:12:26.165826 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:12:26.552460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:26.549511 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:28.074139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:28.074098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:29.602305 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:29.602230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:31.263453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:31.263110 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:31.475515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:31.475480 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:32.088286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:32.088248 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:32.248291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:32.248253 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:33.785629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:33.785568 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:33.940219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:33.940185 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:35.779987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:35.779952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:37.314693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:37.314654 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:38.495176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:38.495124 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:38.646963 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:38.646927 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:39.159018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:39.158983 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:40.903524 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:40.903487 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:41.158481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:41.158368 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:41.334438 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:41.334400 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:41.577637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:41.577604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:41.740143 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:41.740105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:42.358337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:42.358303 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:42.531450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:42.530160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:43.276021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:43.275980 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:44.110592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:44.110561 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:45.278400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:45.278299 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:45.500255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:45.500224 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:46.894197 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:46.894158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:48.502606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:48.502570 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:49.078660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.078628 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:49.225538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.225499 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:49.360812 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.360728 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng"] Nov 03 17:12:49.365500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.365480 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.381620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.381596 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9a501d\"/\"openshift-service-ca.crt\"" Nov 03 17:12:49.391526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.391498 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9a501d\"/\"kube-root-ca.crt\"" Nov 03 17:12:49.391797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.391780 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9a501d\"/\"default-dockercfg-z55zn\"" Nov 03 17:12:49.391965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.391951 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9a501d\"/\"e2e-usage-sim-a801fb-cert\"" Nov 03 17:12:49.396748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.396723 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng"] Nov 03 17:12:49.446023 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.445987 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls\") pod \"e2e-usage-sim-a801fb-f4b99cb69-rbfng\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.446607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.446576 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrpm\" (UniqueName: \"kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm\") pod \"e2e-usage-sim-a801fb-f4b99cb69-rbfng\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.548777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.548740 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls\") pod \"e2e-usage-sim-a801fb-f4b99cb69-rbfng\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.549351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.549330 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrpm\" (UniqueName: \"kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm\") pod \"e2e-usage-sim-a801fb-f4b99cb69-rbfng\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.551939 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.551912 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls\") pod \"e2e-usage-sim-a801fb-f4b99cb69-rbfng\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.571352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.571313 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrpm\" (UniqueName: \"kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm\") pod \"e2e-usage-sim-a801fb-f4b99cb69-rbfng\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.679062 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.678968 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:49.987484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:49.987450 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng"] Nov 03 17:12:49.987913 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:12:49.987888 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24eb9b51_fdc9_45a3_b9c9_e797769f5210.slice/crio-f9b0c61d19e316c0113d698bd6a2f8c1dd13e11811569f592865f935b8cb25df WatchSource:0}: Error finding container f9b0c61d19e316c0113d698bd6a2f8c1dd13e11811569f592865f935b8cb25df: Status 404 returned error can't find the container with id f9b0c61d19e316c0113d698bd6a2f8c1dd13e11811569f592865f935b8cb25df Nov 03 17:12:50.112803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:50.112776 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:50.769137 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:50.769087 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" event={"ID":"24eb9b51-fdc9-45a3-b9c9-e797769f5210","Type":"ContainerStarted","Data":"2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17"} Nov 03 17:12:50.769660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:50.769146 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:50.769660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:50.769165 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" event={"ID":"24eb9b51-fdc9-45a3-b9c9-e797769f5210","Type":"ContainerStarted","Data":"f9b0c61d19e316c0113d698bd6a2f8c1dd13e11811569f592865f935b8cb25df"} Nov 03 17:12:50.774780 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:50.774755 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:12:50.806942 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:50.806880 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" podStartSLOduration=1.8068596270000001 podStartE2EDuration="1.806859627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:50.80573696 +0000 UTC m=+3514.499007981" watchObservedRunningTime="2026-11-03 17:12:50.806859627 +0000 UTC m=+3514.500130651" Nov 03 17:12:51.699988 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:51.695061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:53.276134 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:53.276100 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:54.870720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:54.870687 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:55.343242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:55.343191 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:55.620509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:55.619895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:56.196059 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:12:56.196017 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:12:57.326794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:57.326736 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:12:58.976927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:12:58.976830 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:00.546446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:00.545407 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:00.780585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:00.780547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:02.282093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:02.282065 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:03.842583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:03.842549 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:05.459450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:05.454688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:06.433659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:06.433622 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:07.221970 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:07.221919 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:09.075525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:09.075387 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:10.696534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:10.696498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:12.350995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:12.350961 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:13.871584 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:13.871549 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:14.935490 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:14.935450 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:15.497454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:15.497389 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:17.225846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:17.225813 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:18.962599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:18.962561 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:20.575710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:20.575659 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:21.960137 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:21.960098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:22.289457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:22.289404 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:23.893224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:23.893189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:25.586230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:25.586200 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:26.158858 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:13:26.158813 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:13:27.487450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:27.485969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:29.101642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:29.101544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:30.720416 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:30.720336 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:32.465763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:32.465728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:32.658457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:32.654177 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:33.687241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:33.687206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:33.824669 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:33.824641 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:34.301007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:34.293828 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:34.463273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:34.463228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:36.118574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:36.118541 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:37.807946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:37.807897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:39.374156 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:39.374122 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:40.656971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:40.656933 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:40.792894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:40.792858 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:40.943316 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:40.943228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:41.103926 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:41.103895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:42.605454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:42.602450 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:44.263287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:44.260922 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:45.887101 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:45.887056 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:47.185653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:47.185622 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:47.344364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:47.344322 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:47.508206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:47.508110 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:48.123914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:48.123878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:48.383093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:48.382997 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:50.114254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:50.114221 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:50.384368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:50.384288 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:51.838464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:51.838407 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:53.477966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:53.477928 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:53.678484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:53.674812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:54.014857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:54.014824 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:56.161534 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:13:56.161497 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:13:56.278014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:56.277937 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:58.137527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:58.137418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:13:59.713459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:13:59.713402 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:01.436107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:01.436065 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:01.605231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:01.602971 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:03.675541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:03.675509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:03.853146 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:03.853111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:04.017089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:04.016995 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:04.521396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:04.514191 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:05.940307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:05.940256 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:07.571715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:07.571681 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:09.188199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:09.188158 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:09.643331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:09.643297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:09.893409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:09.893314 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:10.869188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:10.869155 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:12.451048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:12.451017 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:14.086324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:14.086289 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:15.031602 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:15.031553 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:15.792635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:15.792587 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:15.946317 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:15.946280 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:16.100411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:16.100315 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:16.561443 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:16.557976 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:17.176071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:17.176040 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:14:17.188503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:17.188482 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:14:17.693657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:17.693623 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:19.296667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:19.296630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:20.886710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:20.886679 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:22.540447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:22.528638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:23.737978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:23.728496 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:23.913803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:23.913766 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:24.148447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:24.147686 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:24.426509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:24.426337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:24.901971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:24.901931 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:25.061450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:25.060952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:25.239503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:25.238534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:25.405739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:25.405592 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:25.567056 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:25.567023 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:26.269704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:26.257921 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:26.392716 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:26.392672 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.489557 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:26.489508 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.639818 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:14:26.639776 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:14:26.809578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:26.809528 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:28.238888 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:28.238851 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rl675 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:14:28.251553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:28.251503 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:28.285575 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:28.285540 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:28.594810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:28.594777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:29.428941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:29.428907 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:30.687452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:30.669114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:32.330034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:32.330001 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:32.503298 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:32.503251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:32.666149 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:32.666058 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:32.841725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:32.841688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:33.064458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:33.064404 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:33.284792 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:33.284746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:34.011095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:34.011053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:34.172575 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:34.172534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:34.743187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:34.743154 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:35.030193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:35.030145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:36.541869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:36.541826 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:37.802224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:37.802187 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:38.041152 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:38.041120 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:38.446452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:38.438337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:39.087504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:39.087457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:39.491049 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:39.490935 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:39.644923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:39.644860 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:39.997116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:39.997085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:40.149024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:40.148990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:40.378762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:40.378729 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:40.523615 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:40.523580 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:40.926911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:40.926869 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:41.302652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:41.302611 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:42.328684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:42.328646 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:44.022220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:44.022179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:44.789931 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:44.789897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:44.949892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:44.949855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:45.872368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:45.872200 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:46.134877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:46.134791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:46.559272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:46.559241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:46.717792 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:46.717759 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:47.885913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:47.885878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:48.048913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:48.048877 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:48.334130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:48.334098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:49.845368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:49.845331 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:49.988289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:49.988255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:50.334150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:50.334113 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:50.505452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:50.479918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:51.540150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:51.540113 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:53.957857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:53.957795 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:54.171633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:54.171604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:55.135173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:55.135101 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:55.287412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:55.287365 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:55.445507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:55.444414 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:55.603035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:55.599951 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:55.799466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:55.799415 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:56.158946 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:14:56.158863 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:14:56.207483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:56.207385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:56.451566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:56.451471 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:57.934147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:57.934107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:58.192090 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:58.191991 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:14:59.585234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:14:59.585204 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:00.393466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.393413 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv"] Nov 03 17:15:00.398738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.398714 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.420393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.420370 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv"] Nov 03 17:15:00.458686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.458651 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:15:00.460416 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.460388 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.460592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.460445 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntmp\" (UniqueName: \"kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.562215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.562168 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.562418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.562224 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntmp\" (UniqueName: \"kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.562974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.562952 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.585680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.585656 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntmp\" (UniqueName: \"kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:00.738438 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:00.738328 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:01.047732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:01.047692 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv"] Nov 03 17:15:01.051453 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:15:01.051397 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd348df1f_5766_48f0_8918_384cd000e606.slice/crio-c8f13876b040b89e86501157c6a0b5a5da978c87d486a0b11e642cf13114bce8 WatchSource:0}: Error finding container c8f13876b040b89e86501157c6a0b5a5da978c87d486a0b11e642cf13114bce8: Status 404 returned error can't find the container with id c8f13876b040b89e86501157c6a0b5a5da978c87d486a0b11e642cf13114bce8 Nov 03 17:15:01.397463 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:01.389443 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:01.605906 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:01.605794 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" event={"ID":"d348df1f-5766-48f0-8918-384cd000e606","Type":"ContainerStarted","Data":"e70c154580e18d27d438059656fa7499f209f4397d63b49061d5e18f40a76ad9"} Nov 03 17:15:01.605906 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:01.605870 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" event={"ID":"d348df1f-5766-48f0-8918-384cd000e606","Type":"ContainerStarted","Data":"c8f13876b040b89e86501157c6a0b5a5da978c87d486a0b11e642cf13114bce8"} Nov 03 17:15:01.645654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:01.645598 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" podStartSLOduration=1.64558061 podStartE2EDuration="1.64558061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:01.64403922 +0000 UTC m=+3645.337310243" watchObservedRunningTime="2026-11-03 17:15:01.64558061 +0000 UTC m=+3645.338851629" Nov 03 17:15:02.628464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:02.628398 4539 generic.go:356] "Generic (PLEG): container finished" podID="d348df1f-5766-48f0-8918-384cd000e606" containerID="e70c154580e18d27d438059656fa7499f209f4397d63b49061d5e18f40a76ad9" exitCode=60 Nov 03 17:15:02.629268 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:02.628409 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" event={"ID":"d348df1f-5766-48f0-8918-384cd000e606","Type":"ContainerDied","Data":"e70c154580e18d27d438059656fa7499f209f4397d63b49061d5e18f40a76ad9"} Nov 03 17:15:02.629487 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:02.629113 4539 scope.go:122] "RemoveContainer" containerID="e70c154580e18d27d438059656fa7499f209f4397d63b49061d5e18f40a76ad9" Nov 03 17:15:03.050447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.048680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:03.378995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.378960 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:03.545123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.545087 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:03.641834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.641751 4539 generic.go:356] "Generic (PLEG): container finished" podID="d348df1f-5766-48f0-8918-384cd000e606" containerID="c56611b85f124972615ec194e1cf690baf7841df6a0ffd8adcb2724e76e5e16f" exitCode=60 Nov 03 17:15:03.642307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.641889 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" event={"ID":"d348df1f-5766-48f0-8918-384cd000e606","Type":"ContainerDied","Data":"c56611b85f124972615ec194e1cf690baf7841df6a0ffd8adcb2724e76e5e16f"} Nov 03 17:15:03.642307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.641965 4539 scope.go:122] "RemoveContainer" containerID="e70c154580e18d27d438059656fa7499f209f4397d63b49061d5e18f40a76ad9" Nov 03 17:15:03.642406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.642387 4539 scope.go:122] "RemoveContainer" containerID="c56611b85f124972615ec194e1cf690baf7841df6a0ffd8adcb2724e76e5e16f" Nov 03 17:15:03.642954 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:03.642927 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv_odh-ai-gateway-infra(d348df1f-5766-48f0-8918-384cd000e606)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" podUID="d348df1f-5766-48f0-8918-384cd000e606" Nov 03 17:15:03.879754 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:03.879720 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:04.426449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:04.414324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:04.685507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:04.677252 4539 scope.go:122] "RemoveContainer" containerID="c56611b85f124972615ec194e1cf690baf7841df6a0ffd8adcb2724e76e5e16f" Nov 03 17:15:04.685507 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:04.677733 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv_odh-ai-gateway-infra(d348df1f-5766-48f0-8918-384cd000e606)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" podUID="d348df1f-5766-48f0-8918-384cd000e606" Nov 03 17:15:05.044069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:05.044020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:06.772021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:06.767112 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:08.482919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:08.482873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:08.668118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:08.668043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:09.955151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:09.955098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:10.307183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:10.307146 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:10.456915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:10.456873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:12.074725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:12.074680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:13.609450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:13.605198 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:15.186093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:15.186064 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:15.326777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:15.326693 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:16.869750 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:16.869720 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:17.461925 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:17.461897 4539 scope.go:122] "RemoveContainer" containerID="c56611b85f124972615ec194e1cf690baf7841df6a0ffd8adcb2724e76e5e16f" Nov 03 17:15:18.584468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:18.584336 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:18.864390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:18.864301 4539 generic.go:356] "Generic (PLEG): container finished" podID="d348df1f-5766-48f0-8918-384cd000e606" containerID="eebf54204cd61a6cf9001016b3299414635d05bf024b8e51b6d95b7d7dec6dc8" exitCode=60 Nov 03 17:15:18.864390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:18.864371 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" event={"ID":"d348df1f-5766-48f0-8918-384cd000e606","Type":"ContainerDied","Data":"eebf54204cd61a6cf9001016b3299414635d05bf024b8e51b6d95b7d7dec6dc8"} Nov 03 17:15:18.864661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:18.864470 4539 scope.go:122] "RemoveContainer" containerID="c56611b85f124972615ec194e1cf690baf7841df6a0ffd8adcb2724e76e5e16f" Nov 03 17:15:18.864856 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:18.864837 4539 scope.go:122] "RemoveContainer" containerID="eebf54204cd61a6cf9001016b3299414635d05bf024b8e51b6d95b7d7dec6dc8" Nov 03 17:15:18.865159 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:18.865140 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv_odh-ai-gateway-infra(d348df1f-5766-48f0-8918-384cd000e606)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" podUID="d348df1f-5766-48f0-8918-384cd000e606" Nov 03 17:15:19.920170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:19.920138 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv"] Nov 03 17:15:20.095386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.095361 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:20.180018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.179920 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp\" (UniqueName: \"kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp\") pod \"d348df1f-5766-48f0-8918-384cd000e606\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " Nov 03 17:15:20.180018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.179981 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca\") pod \"d348df1f-5766-48f0-8918-384cd000e606\" (UID: \"d348df1f-5766-48f0-8918-384cd000e606\") " Nov 03 17:15:20.180929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.180898 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca" pod "d348df1f-5766-48f0-8918-384cd000e606" (UID: "d348df1f-5766-48f0-8918-384cd000e606"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:20.181199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.181181 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d348df1f-5766-48f0-8918-384cd000e606-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:15:20.183082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.183057 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp" pod "d348df1f-5766-48f0-8918-384cd000e606" (UID: "d348df1f-5766-48f0-8918-384cd000e606"). InnerVolumeSpecName "kube-api-access-sntmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:20.282664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.282615 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sntmp\" (UniqueName: \"kubernetes.io/projected/d348df1f-5766-48f0-8918-384cd000e606-kube-api-access-sntmp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:15:20.304453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.288694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:20.881345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.881313 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" Nov 03 17:15:20.881585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.881308 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv" event={"ID":"d348df1f-5766-48f0-8918-384cd000e606","Type":"ContainerDied","Data":"c8f13876b040b89e86501157c6a0b5a5da978c87d486a0b11e642cf13114bce8"} Nov 03 17:15:20.881585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.881478 4539 scope.go:122] "RemoveContainer" containerID="eebf54204cd61a6cf9001016b3299414635d05bf024b8e51b6d95b7d7dec6dc8" Nov 03 17:15:20.938382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.938345 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv"] Nov 03 17:15:20.948410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:20.948379 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895435-t8nfv"] Nov 03 17:15:21.977227 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:21.977190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:22.469898 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:22.469865 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d348df1f-5766-48f0-8918-384cd000e606" path="/var/lib/kubelet/pods/d348df1f-5766-48f0-8918-384cd000e606/volumes" Nov 03 17:15:23.908665 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:23.908628 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:24.091883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:24.090877 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:24.381603 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:24.381554 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:26.158587 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:26.158552 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:15:26.299894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:26.299733 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:28.079479 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:28.079418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:29.802321 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:29.802277 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:31.396455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:31.381877 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:32.952851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:32.952817 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:33.098552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:33.098524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:33.286117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:33.286048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:34.651158 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:34.651125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:36.312488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:36.312452 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:37.903460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:37.894154 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:39.501448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:39.497544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:41.125449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:41.122815 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:42.561960 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:42.561925 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:42.847712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:42.847598 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:42.993646 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:42.993610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:43.144545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:43.144439 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:44.588471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:44.576106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:46.127495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:46.127457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:47.723453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:47.722903 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:49.398347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:49.398309 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:51.237393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:51.237356 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:53.060911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:53.060875 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:54.888277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:54.888236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:56.170070 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:56.170029 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:15:56.694924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:56.692849 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:58.119877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.119839 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm"] Nov 03 17:15:58.121083 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121056 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.121083 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121084 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.121255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121115 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.121255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121123 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.121332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121306 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.121332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121319 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.121332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.121328 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:15:58.125781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.125761 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.145112 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.145077 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm"] Nov 03 17:15:58.146288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.146260 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cb0fa3-serving-cert\"" Nov 03 17:15:58.147303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.147277 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.147408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.147339 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.147481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.147464 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5czd\" (UniqueName: \"kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.147685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.147667 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.231614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.231582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:58.249636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.249594 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.249817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.249737 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.249817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.249780 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.249933 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.249878 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5czd\" (UniqueName: \"kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.249933 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:58.249915 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-serving-cert: secret "maas-api-e2e-inf-a-cb0fa3-serving-cert" not found Nov 03 17:15:58.250054 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:15:58.249989 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls podName:4c791259-d616-4cd1-8def-c481def1035f nodeName:}" failed. No retries permitted until 2026-11-03 17:15:58.749966881 +0000 UTC m=+3702.443237884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls") pod "maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" (UID: "4c791259-d616-4cd1-8def-c481def1035f") : secret "maas-api-e2e-inf-a-cb0fa3-serving-cert" not found Nov 03 17:15:58.250234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.250213 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.252856 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.252837 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.274534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.274505 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5czd\" (UniqueName: \"kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.385041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.383561 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:58.540086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.540048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:15:58.761770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.761659 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:58.764496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:58.764468 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:59.053641 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:59.053597 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:15:59.381907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:59.381872 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm"] Nov 03 17:15:59.384266 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:15:59.384230 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c791259_d616_4cd1_8def_c481def1035f.slice/crio-86a9411dfd134b5c4f46e037cb730ccadaf597509cd912c1cde49f3d61e664c4 WatchSource:0}: Error finding container 86a9411dfd134b5c4f46e037cb730ccadaf597509cd912c1cde49f3d61e664c4: Status 404 returned error can't find the container with id 86a9411dfd134b5c4f46e037cb730ccadaf597509cd912c1cde49f3d61e664c4 Nov 03 17:15:59.385539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:59.385522 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:15:59.567737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:15:59.567634 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" event={"ID":"4c791259-d616-4cd1-8def-c481def1035f","Type":"ContainerStarted","Data":"86a9411dfd134b5c4f46e037cb730ccadaf597509cd912c1cde49f3d61e664c4"} Nov 03 17:16:00.022532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:00.019692 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:00.577175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:00.577139 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" event={"ID":"4c791259-d616-4cd1-8def-c481def1035f","Type":"ContainerStarted","Data":"f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2"} Nov 03 17:16:00.577673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:00.577204 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:16:00.620747 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:00.620679 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" podStartSLOduration=2.283085233 podStartE2EDuration="2.620658079s" totalImagesPullingTime="337.572846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:00.616609464 +0000 UTC m=+3704.309880493" watchObservedRunningTime="2026-11-03 17:16:00.620658079 +0000 UTC m=+3704.313929101" Nov 03 17:16:01.883830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:01.883799 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:03.704200 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:03.704152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:05.625176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:05.625116 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:06.591117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:06.591090 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:16:07.406916 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:07.406881 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:09.014343 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:09.011166 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:10.713389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:10.713351 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:12.326384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:12.326254 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:13.934658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:13.934615 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:15.264814 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:15.264784 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:15.490125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:15.490092 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:17.413920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:17.413888 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:19.078023 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:19.077990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:20.627989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:20.627943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:20.787507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:20.787476 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:21.129178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:21.129142 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:21.441037 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:21.440944 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:22.594048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:22.594014 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:24.464325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:24.464294 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:24.645502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:24.645398 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:24.821639 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:24.821605 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:25.949857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:25.949819 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:26.386053 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:16:26.386012 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:16:26.689794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:26.689685 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:26.886693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:26.886642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:29.086631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:29.084068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:30.949446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:30.948986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:32.532817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:32.532785 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:32.678276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:32.678241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:33.039113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:33.039069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:33.501514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:33.501104 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:34.289309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:34.289255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:35.970927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:35.970898 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:37.032017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:37.031985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:37.326014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:37.325928 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:37.999392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:37.999235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:38.372370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:38.372335 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:39.839547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:39.839418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:41.149813 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:41.149774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:41.544682 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:41.544645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:41.690460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:41.689140 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:42.013892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:42.013772 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:42.162143 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:42.162095 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:43.493300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:43.493241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:43.757356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:43.757272 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:45.368738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:45.368705 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:46.324584 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:46.324547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:46.501048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:46.501008 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:47.415913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:47.415880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:47.697532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:47.697437 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:48.074242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:48.074201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:48.229742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:48.229711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:49.265276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:49.265241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:50.918279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:50.918245 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:52.607674 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:52.607638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:53.677608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:53.677576 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:53.836150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:53.833755 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:54.249305 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:54.249269 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng"] Nov 03 17:16:54.249564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:54.249536 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" podUID="24eb9b51-fdc9-45a3-b9c9-e797769f5210" containerName="sim" containerID="cri-o://2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17" gracePeriod=30 Nov 03 17:16:54.485978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:54.485935 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:54.688437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:54.688170 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:54.912439 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:54.912389 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:16:55.014852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.014767 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls\" (UniqueName: \"kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls\") pod \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " Nov 03 17:16:55.015015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.014927 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm\" (UniqueName: \"kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm\") pod \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\" (UID: \"24eb9b51-fdc9-45a3-b9c9-e797769f5210\") " Nov 03 17:16:55.017284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.017248 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls" pod "24eb9b51-fdc9-45a3-b9c9-e797769f5210" (UID: "24eb9b51-fdc9-45a3-b9c9-e797769f5210"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:16:55.017284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.017273 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm" pod "24eb9b51-fdc9-45a3-b9c9-e797769f5210" (UID: "24eb9b51-fdc9-45a3-b9c9-e797769f5210"). InnerVolumeSpecName "kube-api-access-msrpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:55.090410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.090367 4539 generic.go:356] "Generic (PLEG): container finished" podID="24eb9b51-fdc9-45a3-b9c9-e797769f5210" containerID="2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17" exitCode=0 Nov 03 17:16:55.090410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.090413 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" Nov 03 17:16:55.090728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.090491 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" event={"ID":"24eb9b51-fdc9-45a3-b9c9-e797769f5210","Type":"ContainerDied","Data":"2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17"} Nov 03 17:16:55.090728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.090537 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng" event={"ID":"24eb9b51-fdc9-45a3-b9c9-e797769f5210","Type":"ContainerDied","Data":"f9b0c61d19e316c0113d698bd6a2f8c1dd13e11811569f592865f935b8cb25df"} Nov 03 17:16:55.090728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.090561 4539 scope.go:122] "RemoveContainer" containerID="2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17" Nov 03 17:16:55.104263 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.104241 4539 scope.go:122] "RemoveContainer" containerID="2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17" Nov 03 17:16:55.104668 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:16:55.104646 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17\": container with ID starting with 2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17 not found: ID does not exist" containerID="2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17" Nov 03 17:16:55.104753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.104679 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17"} err="failed to get container status \"2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17\": rpc error: code = NotFound desc = could not find container \"2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17\": container with ID starting with 2499f6d7b4b56e3266c5551db86d75d4e4741b746c95fa18b6fef1d7e36f0d17 not found: ID does not exist" Nov 03 17:16:55.116454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.116411 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrpm\" (UniqueName: \"kubernetes.io/projected/24eb9b51-fdc9-45a3-b9c9-e797769f5210-kube-api-access-msrpm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:16:55.116454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.116454 4539 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/24eb9b51-fdc9-45a3-b9c9-e797769f5210-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:16:55.157505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.155491 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng"] Nov 03 17:16:55.175489 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:55.164694 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-9a501d/e2e-usage-sim-a801fb-f4b99cb69-rbfng"] Nov 03 17:16:56.163288 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:16:56.163227 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:16:56.283150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.283106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:56.440510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.439894 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:56.561452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.553831 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24eb9b51-fdc9-45a3-b9c9-e797769f5210" path="/var/lib/kubelet/pods/24eb9b51-fdc9-45a3-b9c9-e797769f5210/volumes" Nov 03 17:16:56.603447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.596582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:56.892721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.892682 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.940336 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9"] Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.941793 4539 cpu_manager.go:414] "containerMap: removing container" podUID="24eb9b51-fdc9-45a3-b9c9-e797769f5210" containerName="sim" Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.941815 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24eb9b51-fdc9-45a3-b9c9-e797769f5210" containerName="sim" Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.941852 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.941860 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d348df1f-5766-48f0-8918-384cd000e606" containerName="cleanup" Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.942768 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="24eb9b51-fdc9-45a3-b9c9-e797769f5210" containerName="sim" Nov 03 17:16:56.953220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.950303 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:56.974455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.968541 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-547925-openshift-default-dockercfg-5rrq9\"" Nov 03 17:16:56.996460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:56.996391 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9"] Nov 03 17:16:57.055280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.055246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:57.088501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.088467 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.088829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.088785 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzgk\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.089018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.088998 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.089166 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.089149 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.089548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.089515 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.089713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.089696 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.089911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.089893 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.090074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.090056 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.090323 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.090305 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.191762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.191650 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.191762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.191697 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.191775 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.191820 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.191956 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192010 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192049 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzgk\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192079 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192109 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192276 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192544 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192638 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.192883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192722 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.193283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.192899 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.194848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.194823 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.195323 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.195300 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.203953 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.203881 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:57.214318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.214201 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.215139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.215106 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzgk\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.296468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.292011 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:57.403357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:57.403320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:58.059517 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:16:58.059479 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee28d928_7be1_4218_b50d_f1fd00a37eca.slice/crio-4b530192156239b2fcdb15417648c7cfeb848bfb2426f469b39ad874cb40f6bc WatchSource:0}: Error finding container 4b530192156239b2fcdb15417648c7cfeb848bfb2426f469b39ad874cb40f6bc: Status 404 returned error can't find the container with id 4b530192156239b2fcdb15417648c7cfeb848bfb2426f469b39ad874cb40f6bc Nov 03 17:16:58.061374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:58.061348 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9"] Nov 03 17:16:58.062623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:58.062592 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:16:58.062698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:58.062658 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:16:58.062698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:58.062687 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:16:58.092576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:58.092544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:58.129192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:58.128502 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" event={"ID":"ee28d928-7be1-4218-b50d-f1fd00a37eca","Type":"ContainerStarted","Data":"4b530192156239b2fcdb15417648c7cfeb848bfb2426f469b39ad874cb40f6bc"} Nov 03 17:16:59.035442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:59.035391 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:16:59.138751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:59.138708 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" event={"ID":"ee28d928-7be1-4218-b50d-f1fd00a37eca","Type":"ContainerStarted","Data":"61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5"} Nov 03 17:16:59.204619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:59.204561 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podStartSLOduration=3.204545028 podStartE2EDuration="3.204545028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:59.19156936 +0000 UTC m=+3762.884840406" watchObservedRunningTime="2026-11-03 17:16:59.204545028 +0000 UTC m=+3762.897816050" Nov 03 17:16:59.297344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:59.297249 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:16:59.418948 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:16:59.406914 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:17:00.149495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:00.149466 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:17:00.150498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:00.150472 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:17:00.854997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:00.854953 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:01.707546 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:01.707515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:01.916585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:01.916538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:02.547337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:02.547305 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:03.045690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:03.045637 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:03.676076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:03.676020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:03.838117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:03.838077 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:04.336296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:04.336258 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:04.490911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:04.490879 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:04.696077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:04.695996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:04.843006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:04.842970 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:05.554195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:05.554151 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:06.065191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:06.065160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:06.716306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:06.716210 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:08.232457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:08.230493 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:09.988659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:09.988615 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:10.174453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:10.174219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:11.333646 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:11.333609 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:11.841606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:11.841569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:12.015486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:12.015412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:13.781221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:13.781190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:15.385854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:15.385809 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:15.529866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:15.529833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:16.995277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:16.995241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:18.655161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:18.651757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:20.220841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:20.220806 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:20.378773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:20.378611 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:22.117982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:22.116908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:23.695845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:23.695804 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:25.284816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:25.284774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:26.157959 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:17:26.157923 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:17:26.794074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:26.794041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:28.826744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:28.826692 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:30.477856 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:30.477812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:32.028925 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.028873 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv"] Nov 03 17:17:32.034499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.034481 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.052797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.052729 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-236539-serving-cert\"" Nov 03 17:17:32.056251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.056204 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv"] Nov 03 17:17:32.069637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.069262 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.069637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.069404 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.070190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.069863 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9859p\" (UniqueName: \"kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.070190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.070013 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.151725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.151660 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:32.171406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.171377 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.171578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.171513 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.171578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.171544 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9859p\" (UniqueName: \"kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.171704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.171663 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.171818 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:17:32.171793 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-serving-cert: secret "maas-api-e2e-inf-b-236539-serving-cert" not found Nov 03 17:17:32.171909 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:17:32.171893 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls podName:305e93ab-f9a0-4374-80a0-8c533b7684d4 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:32.671866285 +0000 UTC m=+3796.365137287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls") pod "maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" (UID: "305e93ab-f9a0-4374-80a0-8c533b7684d4") : secret "maas-api-e2e-inf-b-236539-serving-cert" not found Nov 03 17:17:32.172205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.172182 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.174160 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.174143 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.195886 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.195860 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9859p\" (UniqueName: \"kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.300538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.299912 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:32.445169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.445142 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:32.680248 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.680159 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.682869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.682843 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:32.953173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:32.953073 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:33.325789 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:17:33.325757 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305e93ab_f9a0_4374_80a0_8c533b7684d4.slice/crio-7e8c9b9b9637db33c17138ac981d194ae2b474f9ed318276cc1764c759ab1787 WatchSource:0}: Error finding container 7e8c9b9b9637db33c17138ac981d194ae2b474f9ed318276cc1764c759ab1787: Status 404 returned error can't find the container with id 7e8c9b9b9637db33c17138ac981d194ae2b474f9ed318276cc1764c759ab1787 Nov 03 17:17:33.326227 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:33.325976 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv"] Nov 03 17:17:33.599990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:33.599904 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" event={"ID":"305e93ab-f9a0-4374-80a0-8c533b7684d4","Type":"ContainerStarted","Data":"7e8c9b9b9637db33c17138ac981d194ae2b474f9ed318276cc1764c759ab1787"} Nov 03 17:17:33.977802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:33.977725 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:34.609531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:34.609495 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" event={"ID":"305e93ab-f9a0-4374-80a0-8c533b7684d4","Type":"ContainerStarted","Data":"0239b6e6952af758fa043c400c631adefd9000532b1f8301b004042e2e9c21cb"} Nov 03 17:17:34.609907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:34.609565 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:34.655360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:34.655304 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" podStartSLOduration=3.30781528 podStartE2EDuration="3.655290262s" totalImagesPullingTime="347.474982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:34.653191147 +0000 UTC m=+3798.346462170" watchObservedRunningTime="2026-11-03 17:17:34.655290262 +0000 UTC m=+3798.348561284" Nov 03 17:17:35.544558 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:35.544522 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:37.141059 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:37.141019 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:38.871293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:38.871177 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:40.409920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:40.409888 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:40.632129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:40.632093 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:17:42.062774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:42.062740 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:42.208833 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:42.208800 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:42.352279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:42.352193 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:43.675659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:43.675627 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:45.199832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:45.199798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:46.873673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:46.873634 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:48.406463 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:48.406415 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:49.982961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:49.982923 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:51.569093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:51.569059 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:53.496371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:53.495229 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:55.073624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:55.064971 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:56.162387 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:17:56.162348 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:17:56.640793 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:56.640759 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:56.800136 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:56.800067 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:56.952993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:56.952886 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:17:58.449452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:17:58.445253 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:00.093094 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:00.093053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:01.700458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:01.700268 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:03.294015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:03.293982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:04.945449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:04.927477 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:06.421510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.421472 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc"] Nov 03 17:18:06.426149 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.426125 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.449680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.449652 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-472bee-serving-cert\"" Nov 03 17:18:06.457217 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.457190 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc"] Nov 03 17:18:06.478711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.478603 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.478711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.478667 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwqj\" (UniqueName: \"kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.478961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.478786 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.478961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.478840 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.547538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.547506 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:06.580747 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.580719 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.580954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.580761 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwqj\" (UniqueName: \"kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.580954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.580825 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.580954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.580857 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.581339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.581308 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.583385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.583358 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.583543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.583522 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.689397 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.689232 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwqj\" (UniqueName: \"kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj\") pod \"maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:06.701849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.701812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:06.740661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:06.740626 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:07.056038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:07.056000 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc"] Nov 03 17:18:07.056502 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:18:07.056478 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c73247_c9ce_408e_8569_d1eb2eaaa77d.slice/crio-d8fb92a065a2ffb06bea04c12500ce9c8ad0e5122bd237bf11bb5f7444994a62 WatchSource:0}: Error finding container d8fb92a065a2ffb06bea04c12500ce9c8ad0e5122bd237bf11bb5f7444994a62: Status 404 returned error can't find the container with id d8fb92a065a2ffb06bea04c12500ce9c8ad0e5122bd237bf11bb5f7444994a62 Nov 03 17:18:07.989277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:07.989245 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" event={"ID":"41c73247-c9ce-408e-8569-d1eb2eaaa77d","Type":"ContainerStarted","Data":"a89d2a58cdeda8ae14c994e2f3a95dd9fd229701a75769d19ce7c5c2a32f6155"} Nov 03 17:18:07.989769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:07.989307 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" event={"ID":"41c73247-c9ce-408e-8569-d1eb2eaaa77d","Type":"ContainerStarted","Data":"d8fb92a065a2ffb06bea04c12500ce9c8ad0e5122bd237bf11bb5f7444994a62"} Nov 03 17:18:07.989769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:07.989339 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:08.037057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:08.036998 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" podStartSLOduration=1.657218241 podStartE2EDuration="2.036980755s" totalImagesPullingTime="379.762514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:08.029318057 +0000 UTC m=+3831.722589080" watchObservedRunningTime="2026-11-03 17:18:08.036980755 +0000 UTC m=+3831.730251793" Nov 03 17:18:08.385245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:08.385216 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:10.086373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:10.086337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:11.739292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:11.739188 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:13.319562 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:13.319018 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:14.004354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:14.004323 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:18:14.885345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:14.885311 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:15.498069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:15.498034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:16.390715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:16.390677 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:18.043848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:18.043812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:19.873079 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:19.873042 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:21.557092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:21.557022 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:23.166245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:23.166204 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:24.790312 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:24.790274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:25.167036 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.166934 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg"] Nov 03 17:18:25.172568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.172539 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.177550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.177514 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg"] Nov 03 17:18:25.195636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.195608 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-24cb6d\"/\"test-model-24cb6d-kserve-self-signed-certs\"" Nov 03 17:18:25.195840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.195619 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-24cb6d\"/\"openshift-service-ca.crt\"" Nov 03 17:18:25.196008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.195615 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-24cb6d\"/\"default-dockercfg-hm782\"" Nov 03 17:18:25.196089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.195715 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-24cb6d\"/\"kube-root-ca.crt\"" Nov 03 17:18:25.233335 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.233286 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:25.265816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.265781 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.266028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.265840 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.266028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.265970 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.266154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.266100 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dpf\" (UniqueName: \"kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.266221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.266149 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.266221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.266211 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.367827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.367786 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368054 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.367859 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368054 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.367951 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dpf\" (UniqueName: \"kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368054 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.368003 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368054 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.368050 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368299 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.368259 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.368402 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.368741 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.368954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.368761 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.370856 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.370834 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.371246 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.371230 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.401447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.401317 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dpf\" (UniqueName: \"kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf\") pod \"test-model-24cb6d-kserve-f9456769c-j7fqg\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.495111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.493819 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:25.940010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:25.939977 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg"] Nov 03 17:18:25.941631 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:18:25.941599 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2309506_25e4_4d6b_a718_11a3a5991386.slice/crio-b5cfa19b8bc6f92a91fb4741135dc385057031772d90d3a2dbbb2bc7ec4c9069 WatchSource:0}: Error finding container b5cfa19b8bc6f92a91fb4741135dc385057031772d90d3a2dbbb2bc7ec4c9069: Status 404 returned error can't find the container with id b5cfa19b8bc6f92a91fb4741135dc385057031772d90d3a2dbbb2bc7ec4c9069 Nov 03 17:18:26.160141 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:18:26.160102 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:18:26.166110 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:26.166077 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" event={"ID":"f2309506-25e4-4d6b-a718-11a3a5991386","Type":"ContainerStarted","Data":"883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5"} Nov 03 17:18:26.166281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:26.166125 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" event={"ID":"f2309506-25e4-4d6b-a718-11a3a5991386","Type":"ContainerStarted","Data":"b5cfa19b8bc6f92a91fb4741135dc385057031772d90d3a2dbbb2bc7ec4c9069"} Nov 03 17:18:26.210931 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:26.210831 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" podStartSLOduration=1.21081392 podStartE2EDuration="1.21081392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:26.205385062 +0000 UTC m=+3849.898656062" watchObservedRunningTime="2026-11-03 17:18:26.21081392 +0000 UTC m=+3849.904084942" Nov 03 17:18:26.494536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:26.494414 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:26.629453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:26.628855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:28.324452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:28.317301 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:29.883961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:29.883924 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:31.620186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:31.620145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:32.769609 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.769569 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq"] Nov 03 17:18:32.774657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.774630 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.798269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.798240 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-527051\"/\"kube-root-ca.crt\"" Nov 03 17:18:32.798847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.798826 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-527051\"/\"default-dockercfg-gdzxz\"" Nov 03 17:18:32.800537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.800503 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-527051\"/\"openshift-service-ca.crt\"" Nov 03 17:18:32.801279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.801254 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-527051\"/\"test-model-527051-kserve-self-signed-certs\"" Nov 03 17:18:32.815680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.812683 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq"] Nov 03 17:18:32.851016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.850964 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.851333 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.851034 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6wk\" (UniqueName: \"kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.851333 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.851252 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.851531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.851474 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.851629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.851589 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.851726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.851650 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.868287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.868247 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:32.953530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.953490 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.953855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.953826 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6wk\" (UniqueName: \"kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.953990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.953930 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.954406 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.954669 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.954798 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.954849 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.955465 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.955519 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.957830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.956819 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.958341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.957928 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:32.976189 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:32.976125 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6wk\" (UniqueName: \"kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk\") pod \"test-model-527051-kserve-75959dd887-sm9lq\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:33.093449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:33.091719 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:33.563399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:33.563362 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq"] Nov 03 17:18:33.563737 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:18:33.563712 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9da67c33_2e77_4f6b_8897_ab346420a222.slice/crio-b97f3fe004b4b3e1fdd8284deb49949989ca6001d14c7b97caee3aeee08d7228 WatchSource:0}: Error finding container b97f3fe004b4b3e1fdd8284deb49949989ca6001d14c7b97caee3aeee08d7228: Status 404 returned error can't find the container with id b97f3fe004b4b3e1fdd8284deb49949989ca6001d14c7b97caee3aeee08d7228 Nov 03 17:18:33.564118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:33.564097 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:34.265895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:34.265849 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" event={"ID":"9da67c33-2e77-4f6b-8897-ab346420a222","Type":"ContainerStarted","Data":"97dc3d96e9fba8b11b16c1982b10dac17a2549853ad9444a56c7902aad4c0d15"} Nov 03 17:18:34.266344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:34.265911 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" event={"ID":"9da67c33-2e77-4f6b-8897-ab346420a222","Type":"ContainerStarted","Data":"b97f3fe004b4b3e1fdd8284deb49949989ca6001d14c7b97caee3aeee08d7228"} Nov 03 17:18:35.097208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:35.096659 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:35.209126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:35.205928 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:35.494762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:35.494663 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:35.552317 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:35.552282 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:35.594371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:35.594302 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" podStartSLOduration=3.594284459 podStartE2EDuration="3.594284459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:34.33631071 +0000 UTC m=+3858.029581767" watchObservedRunningTime="2026-11-03 17:18:35.594284459 +0000 UTC m=+3859.287555480" Nov 03 17:18:36.349405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:36.349369 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:18:36.863270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:36.863232 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:38.468071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:38.468036 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:40.079087 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:40.079040 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:41.655460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:41.653495 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:43.093128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.093095 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:43.113186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.113148 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:43.451490 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.451414 4539 generic.go:356] "Generic (PLEG): container finished" podID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerID="61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5" exitCode=0 Nov 03 17:18:43.451710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.451491 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" event={"ID":"ee28d928-7be1-4218-b50d-f1fd00a37eca","Type":"ContainerDied","Data":"61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5"} Nov 03 17:18:43.451985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.451964 4539 scope.go:122] "RemoveContainer" containerID="61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5" Nov 03 17:18:43.454254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.454227 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:18:43.454354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.454302 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:18:43.454354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.454337 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:18:43.466607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.466578 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:18:43.593726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:43.593693 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:44.484529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:44.484492 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" event={"ID":"ee28d928-7be1-4218-b50d-f1fd00a37eca","Type":"ContainerStarted","Data":"cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa"} Nov 03 17:18:45.150254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:45.150221 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:18:45.156667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:45.156629 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:45.296802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:45.296767 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:18:45.298166 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:45.298136 4539 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:18:45.298342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:45.298194 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 03 17:18:46.297933 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:46.297892 4539 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:18:46.298616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:46.297958 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 03 17:18:46.785910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:46.785873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:47.297060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:47.297028 4539 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:18:47.297258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:47.297087 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 03 17:18:48.297204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:48.297169 4539 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:18:48.297660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:48.297217 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 03 17:18:48.383419 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:48.383378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:49.296859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:49.296823 4539 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:18:49.297073 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:49.296881 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 03 17:18:50.003545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:49.998345 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:50.297529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:50.297491 4539 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:18:50.297760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:50.297542 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 03 17:18:51.302979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:51.302944 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:18:51.304233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:51.304189 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:18:51.488113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:51.488079 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:53.028705 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:53.028658 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:54.537878 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:54.537842 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:56.184165 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:18:56.184130 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:18:56.185818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.185791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:56.384503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.384473 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629"] Nov 03 17:18:56.390071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.390048 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.419491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.419445 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629"] Nov 03 17:18:56.420414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.420357 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-44736d\"/\"openshift-service-ca.crt\"" Nov 03 17:18:56.420659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.420552 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-44736d\"/\"default-dockercfg-vxddc\"" Nov 03 17:18:56.420901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.420866 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-44736d\"/\"kube-root-ca.crt\"" Nov 03 17:18:56.421138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.421107 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-44736d\"/\"test-model-44736d-kserve-self-signed-certs\"" Nov 03 17:18:56.478104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.478020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:56.544720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.544672 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.544929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.544741 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.544929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.544888 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.544929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.544913 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.545113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.545040 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.545184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.545134 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.650891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.650186 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.650891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.650264 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.650891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.650361 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.650891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.650383 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.650891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.650472 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.650891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.650538 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.651390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.651149 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.661448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.652765 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.661448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.653089 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.713630 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.713596 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.734522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.734409 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:56.737774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:56.737737 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm\") pod \"test-model-44736d-kserve-67cbfc6fc5-5p629\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:57.009746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:57.009648 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:57.336659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:57.336460 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629"] Nov 03 17:18:57.337084 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:18:57.336779 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3667ce2_6425_4264_b6bc_592f6272b109.slice/crio-cbdd26da00e0fef82224f2396077e3653f3c428e6d4c89fa3a3902351d260d5a WatchSource:0}: Error finding container cbdd26da00e0fef82224f2396077e3653f3c428e6d4c89fa3a3902351d260d5a: Status 404 returned error can't find the container with id cbdd26da00e0fef82224f2396077e3653f3c428e6d4c89fa3a3902351d260d5a Nov 03 17:18:57.812256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:57.812204 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" event={"ID":"a3667ce2-6425-4264-b6bc-592f6272b109","Type":"ContainerStarted","Data":"feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff"} Nov 03 17:18:57.812654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:57.812630 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" event={"ID":"a3667ce2-6425-4264-b6bc-592f6272b109","Type":"ContainerStarted","Data":"cbdd26da00e0fef82224f2396077e3653f3c428e6d4c89fa3a3902351d260d5a"} Nov 03 17:18:57.868451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:57.864570 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" podStartSLOduration=1.864551861 podStartE2EDuration="1.864551861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:57.862679354 +0000 UTC m=+3881.555950375" watchObservedRunningTime="2026-11-03 17:18:57.864551861 +0000 UTC m=+3881.557822883" Nov 03 17:18:58.011255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:58.011223 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:18:58.302590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:58.302530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:18:59.935910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:18:59.935852 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:00.117254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.117218 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq"] Nov 03 17:19:00.127183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.123453 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.158483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.149398 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq"] Nov 03 17:19:00.160846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.160735 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-236539\"/\"default-dockercfg-95q2m\"" Nov 03 17:19:00.161234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.161209 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-236539\"/\"test-model-236539-kserve-self-signed-certs\"" Nov 03 17:19:00.161666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.161581 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-236539\"/\"openshift-service-ca.crt\"" Nov 03 17:19:00.161820 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.161639 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-236539\"/\"kube-root-ca.crt\"" Nov 03 17:19:00.208093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.207953 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:00.212846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.212815 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.213155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.213134 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.213657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.213402 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.213870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.213849 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892rw\" (UniqueName: \"kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.214245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.214041 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.214472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.214444 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316299 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316416 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892rw\" (UniqueName: \"kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316619 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316701 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316799 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316921 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.316966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.316944 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.317493 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.317468 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.318082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.318058 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.323827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.320223 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.328059 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.328033 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.356450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.354382 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-892rw\" (UniqueName: \"kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw\") pod \"test-model-236539-kserve-96f68f49f-r9slq\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.464506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.463930 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:00.945072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:00.943654 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq"] Nov 03 17:19:00.947413 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:19:00.947373 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod428f9575_cc5c_4cb1_87b8_cc851778cdea.slice/crio-6ec856d81637cbde6a65d2f470b29997a2f4b036225c5f54bf5042070681ba86 WatchSource:0}: Error finding container 6ec856d81637cbde6a65d2f470b29997a2f4b036225c5f54bf5042070681ba86: Status 404 returned error can't find the container with id 6ec856d81637cbde6a65d2f470b29997a2f4b036225c5f54bf5042070681ba86 Nov 03 17:19:01.690729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:01.690693 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:01.871735 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:01.871700 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" event={"ID":"428f9575-cc5c-4cb1-87b8-cc851778cdea","Type":"ContainerStarted","Data":"b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3"} Nov 03 17:19:01.871921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:01.871743 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" event={"ID":"428f9575-cc5c-4cb1-87b8-cc851778cdea","Type":"ContainerStarted","Data":"6ec856d81637cbde6a65d2f470b29997a2f4b036225c5f54bf5042070681ba86"} Nov 03 17:19:01.912497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:01.912392 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" podStartSLOduration=1.91236998 podStartE2EDuration="1.91236998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:01.910929746 +0000 UTC m=+3885.604200769" watchObservedRunningTime="2026-11-03 17:19:01.91236998 +0000 UTC m=+3885.605641001" Nov 03 17:19:02.472456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:02.472406 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:03.156667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.156631 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:03.608230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.608183 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc"] Nov 03 17:19:03.614400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.614369 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.653407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.653380 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc"] Nov 03 17:19:03.658484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.657363 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fc40cc\"/\"openshift-service-ca.crt\"" Nov 03 17:19:03.658484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.658094 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fc40cc\"/\"test-model-fc40cc-kserve-self-signed-certs\"" Nov 03 17:19:03.658484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.658210 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fc40cc\"/\"default-dockercfg-kdtgx\"" Nov 03 17:19:03.658484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.658278 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fc40cc\"/\"kube-root-ca.crt\"" Nov 03 17:19:03.681611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.681578 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:03.697697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.697665 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l5p\" (UniqueName: \"kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.697895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.697723 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.697895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.697796 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.697895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.697851 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.698075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.697999 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.698181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.698165 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.799890 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l5p\" (UniqueName: \"kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.799971 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800012 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800091 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800186 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800334 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800628 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800843 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.801448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.800897 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.804128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.804090 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.812798 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.812271 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:03.822007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.821973 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l5p\" (UniqueName: \"kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p\") pod \"test-model-fc40cc-kserve-8458d96ff7-fnfzc\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:04.035510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:03.992005 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:04.415143 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:19:04.415096 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dd9deaa_49be_411d_9b7d_7d9722f9918f.slice/crio-b6ae971a5d3cc45c14f53e460c0977f253638523d2b9dce0bd9f711322a54361 WatchSource:0}: Error finding container b6ae971a5d3cc45c14f53e460c0977f253638523d2b9dce0bd9f711322a54361: Status 404 returned error can't find the container with id b6ae971a5d3cc45c14f53e460c0977f253638523d2b9dce0bd9f711322a54361 Nov 03 17:19:04.415594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:04.415569 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc"] Nov 03 17:19:04.781824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:04.781790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:05.100267 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:05.100163 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" event={"ID":"0dd9deaa-49be-411d-9b7d-7d9722f9918f","Type":"ContainerStarted","Data":"f0b81fd623adf1570b85c6e42ad359b863e403950dbdb26b1c067d71eeee9fce"} Nov 03 17:19:05.100267 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:05.100203 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" event={"ID":"0dd9deaa-49be-411d-9b7d-7d9722f9918f","Type":"ContainerStarted","Data":"b6ae971a5d3cc45c14f53e460c0977f253638523d2b9dce0bd9f711322a54361"} Nov 03 17:19:05.139017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:05.138964 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" podStartSLOduration=2.138933637 podStartE2EDuration="2.138933637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:05.138395827 +0000 UTC m=+3888.831666849" watchObservedRunningTime="2026-11-03 17:19:05.138933637 +0000 UTC m=+3888.832204657" Nov 03 17:19:06.007390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:06.007350 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:06.678253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:06.678217 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:07.010087 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:07.010004 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:19:07.024165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:07.024139 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:19:07.129553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:07.129516 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:19:08.088614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:08.088574 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:08.381382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:08.381281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:08.963501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:08.962996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:09.114875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:09.114331 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:10.174970 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:10.174929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:10.473186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:10.473103 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:10.478594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:10.478571 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:11.176727 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:11.176697 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:19:11.788694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:11.788656 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:12.969597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:12.969523 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:13.112564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:13.112531 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:13.615846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:13.615812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:13.993273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:13.993187 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:14.008742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:14.008710 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:14.086109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:14.086078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:14.214847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:14.214814 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:19:14.278018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:14.277983 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:15.387332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:15.387304 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:15.532480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:15.532440 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:15.857954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:15.856915 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:17.110186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:17.110138 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:17.255118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:17.255060 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:19:17.270130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:17.270105 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:19:18.684984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:18.684947 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:20.327628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:20.327549 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:21.728756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:21.728716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:21.875746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:21.875710 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:23.420534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:23.417465 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:25.163784 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:25.163709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:26.161532 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:19:26.161488 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:19:26.913666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:26.913634 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:28.565513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:28.565399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:30.263300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:30.263250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:32.038894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:32.038852 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:33.562808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:33.562773 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:34.394431 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:34.394397 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:35.315822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:35.315774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:36.964308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:36.964206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:38.551452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:38.540192 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:40.187024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:40.186982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:41.866987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:41.866954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:43.427489 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:43.427382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:44.936002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:44.935968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:46.509450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:46.506241 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:48.098606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:48.098569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:49.801115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:49.801066 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:51.470626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:51.470597 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:52.978969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:52.978938 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:53.378199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:53.378163 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6"] Nov 03 17:19:53.378408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:53.378385 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" containerID="cri-o://cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7" gracePeriod=30 Nov 03 17:19:53.907365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:53.907325 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92"] Nov 03 17:19:53.907643 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:53.907616 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" containerID="cri-o://60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711" gracePeriod=30 Nov 03 17:19:54.638350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:54.638314 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:56.159661 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:19:56.159621 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:19:56.174598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:56.174565 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:56.480155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:56.480075 4539 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:19:56.480155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:56.480132 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:19:57.803371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:57.803332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:58.674525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.674497 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:19:58.684924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.684888 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.685062 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.684985 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.685219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685187 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:58.685271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685218 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.685271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685251 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:19:58.685491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685469 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:58.685903 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685879 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.685989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685907 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.685989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.685917 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.703439 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.703393 4539 generic.go:356] "Generic (PLEG): container finished" podID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerID="cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7" exitCode=0 Nov 03 17:19:58.703579 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.703459 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" Nov 03 17:19:58.703579 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.703485 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" event={"ID":"e398ba67-40c5-4c34-9cc0-5c5788be6e0f","Type":"ContainerDied","Data":"cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7"} Nov 03 17:19:58.703579 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.703554 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6" event={"ID":"e398ba67-40c5-4c34-9cc0-5c5788be6e0f","Type":"ContainerDied","Data":"1b2525d7a222e83b8ee00bf80671f16bf7f4cdb83317e05de4c95d7ece52b8ce"} Nov 03 17:19:58.703702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.703586 4539 scope.go:122] "RemoveContainer" containerID="cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7" Nov 03 17:19:58.717597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.717578 4539 scope.go:122] "RemoveContainer" containerID="cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7" Nov 03 17:19:58.717951 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:19:58.717923 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7\": container with ID starting with cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7 not found: ID does not exist" containerID="cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7" Nov 03 17:19:58.718050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.717956 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7"} err="failed to get container status \"cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7\": rpc error: code = NotFound desc = could not find container \"cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7\": container with ID starting with cfe1b417092630d5a40966d85d0c1955278ec629871537e2359c3b44e163d3e7 not found: ID does not exist" Nov 03 17:19:58.787317 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787283 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.787551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787325 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.787551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787369 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.787736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787706 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:58.787808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787713 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.788819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787852 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.788819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787929 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:58.788819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.787968 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy\") pod \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\" (UID: \"e398ba67-40c5-4c34-9cc0-5c5788be6e0f\") " Nov 03 17:19:58.788819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.788772 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.788819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.788794 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.790374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.790347 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:19:58.790589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.790562 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:19:58.790775 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.790750 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "kube-api-access-49l5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:19:58.791703 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.791672 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy" pod "e398ba67-40c5-4c34-9cc0-5c5788be6e0f" (UID: "e398ba67-40c5-4c34-9cc0-5c5788be6e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:58.890214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.890182 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49l5z\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-kube-api-access-49l5z\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.890214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.890212 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.890214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.890221 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:58.890742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:58.890229 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398ba67-40c5-4c34-9cc0-5c5788be6e0f-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.054028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.053989 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6"] Nov 03 17:19:59.061455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.061409 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-jwbq6"] Nov 03 17:19:59.188082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.188009 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 17:19:59.295745 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.295714 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.295966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.295771 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.295966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.295865 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.295966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.295927 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.296171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.295968 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.296171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296064 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.296171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296146 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.296351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296202 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.296576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296542 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:59.296679 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296589 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:59.296915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296891 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:59.296996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296921 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:19:59.296996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296977 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:59.297231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.296322 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data\") pod \"94f19163-1b3d-4252-a39e-95d30edfb352\" (UID: \"94f19163-1b3d-4252-a39e-95d30edfb352\") " Nov 03 17:19:59.297987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.297960 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.298085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.297991 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f19163-1b3d-4252-a39e-95d30edfb352-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.298085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.298008 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.298085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.298023 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.298085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.298037 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.298616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.298582 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:19:59.299403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.299383 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:19:59.299941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.299917 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "kube-api-access-9t5jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:19:59.300098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.300016 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token" pod "94f19163-1b3d-4252-a39e-95d30edfb352" (UID: "94f19163-1b3d-4252-a39e-95d30edfb352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:19:59.399380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.399341 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f19163-1b3d-4252-a39e-95d30edfb352-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.399380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.399375 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5jf\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-kube-api-access-9t5jf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.399380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.399388 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f19163-1b3d-4252-a39e-95d30edfb352-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.399713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.399399 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f19163-1b3d-4252-a39e-95d30edfb352-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:19:59.466527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.466416 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:59.611829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.611797 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:19:59.716801 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.716712 4539 generic.go:356] "Generic (PLEG): container finished" podID="94f19163-1b3d-4252-a39e-95d30edfb352" containerID="60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711" exitCode=0 Nov 03 17:19:59.716801 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.716765 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" Nov 03 17:19:59.716993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.716764 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" event={"ID":"94f19163-1b3d-4252-a39e-95d30edfb352","Type":"ContainerDied","Data":"60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711"} Nov 03 17:19:59.716993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.716890 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92" event={"ID":"94f19163-1b3d-4252-a39e-95d30edfb352","Type":"ContainerDied","Data":"6928fe3cfea132513c787b3e216eafa5b993589bc7dd6af16fe8fcef850e2b41"} Nov 03 17:19:59.716993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.716931 4539 scope.go:122] "RemoveContainer" containerID="60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711" Nov 03 17:19:59.731559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.731535 4539 scope.go:122] "RemoveContainer" containerID="60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711" Nov 03 17:19:59.732036 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:19:59.732006 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711\": container with ID starting with 60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711 not found: ID does not exist" containerID="60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711" Nov 03 17:19:59.732103 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.732051 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711"} err="failed to get container status \"60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711\": rpc error: code = NotFound desc = could not find container \"60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711\": container with ID starting with 60e9389e11cd31d455b6314e8f0643ae50e322ee8b10015dfd29b33a4a58b711 not found: ID does not exist" Nov 03 17:19:59.765727 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.765696 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92"] Nov 03 17:19:59.776368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:19:59.776336 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-x2g92"] Nov 03 17:20:00.471239 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:00.471199 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" path="/var/lib/kubelet/pods/94f19163-1b3d-4252-a39e-95d30edfb352/volumes" Nov 03 17:20:00.471985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:00.471960 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" path="/var/lib/kubelet/pods/e398ba67-40c5-4c34-9cc0-5c5788be6e0f/volumes" Nov 03 17:20:01.100671 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:01.100637 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:02.718707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:02.718559 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:04.776893 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:04.776858 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:06.418616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:06.418582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:07.211795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:07.211757 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4"] Nov 03 17:20:07.212350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:07.212316 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" containerID="cri-o://7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca" gracePeriod=30 Nov 03 17:20:08.039483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:08.039378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:08.168753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:08.168715 4539 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:20:08.168941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:08.168773 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:20:08.193536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:08.193502 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:09.623276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.623230 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42"] Nov 03 17:20:09.624013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.623995 4539 cpu_manager.go:414] "containerMap: removing container" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" Nov 03 17:20:09.624013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.624012 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" Nov 03 17:20:09.624163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.624049 4539 cpu_manager.go:414] "containerMap: removing container" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" Nov 03 17:20:09.624163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.624055 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" Nov 03 17:20:09.624163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.624157 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f19163-1b3d-4252-a39e-95d30edfb352" containerName="istio-proxy" Nov 03 17:20:09.624332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.624169 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="e398ba67-40c5-4c34-9cc0-5c5788be6e0f" containerName="istio-proxy" Nov 03 17:20:09.629581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.629558 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.648478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.647255 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-89df9a-serving-cert\"" Nov 03 17:20:09.662633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.662597 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42"] Nov 03 17:20:09.712960 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.712912 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.713204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.713080 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5rt\" (UniqueName: \"kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.713204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.713182 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.713348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.713291 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.736393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.736352 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:09.831451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.819901 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.855449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.854650 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.855449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.854921 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5rt\" (UniqueName: \"kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.855449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.855069 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.856877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.856040 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.856877 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:20:09.856162 4539 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-serving-cert: secret "maas-api-e2e-inf-b-89df9a-serving-cert" not found Nov 03 17:20:09.856877 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:20:09.856242 4539 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls podName:7d6bf843-67e8-4ccf-accf-816501f126fb nodeName:}" failed. No retries permitted until 2026-11-03 17:20:10.356223305 +0000 UTC m=+3954.049494323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls") pod "maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" (UID: "7d6bf843-67e8-4ccf-accf-816501f126fb") : secret "maas-api-e2e-inf-b-89df9a-serving-cert" not found Nov 03 17:20:09.857223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.857143 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.883077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.882992 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5rt\" (UniqueName: \"kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:09.886215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:09.886180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:10.059755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:10.059724 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:10.363850 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:10.363813 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:10.370499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:10.367655 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:10.552144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:10.552096 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:10.909510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:10.907138 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42"] Nov 03 17:20:11.487735 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:11.487705 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:11.835783 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:11.835749 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7"] Nov 03 17:20:11.836028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:11.836002 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" containerID="cri-o://018f86b08eaf1d832c7b72cf32447a7e27fe4d3cf5f8c0833adadf6ebb9c5414" gracePeriod=30 Nov 03 17:20:11.941408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:11.941368 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" event={"ID":"7d6bf843-67e8-4ccf-accf-816501f126fb","Type":"ContainerStarted","Data":"0e7b96bc8e8806d5855309b8885f7c86b58072c31005076270cf935e5d0565f5"} Nov 03 17:20:11.941408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:11.941452 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:11.942005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:11.941473 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" event={"ID":"7d6bf843-67e8-4ccf-accf-816501f126fb","Type":"ContainerStarted","Data":"ec4d3d265072a97c46bd3bdeffaa7079d4437dea7814be989b886ab4e98ef426"} Nov 03 17:20:12.539211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.539179 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 17:20:12.593531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.593480 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" podStartSLOduration=3.170576431 podStartE2EDuration="3.593464073s" totalImagesPullingTime="422.887642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:20:11.987796544 +0000 UTC m=+3955.681067566" watchObservedRunningTime="2026-11-03 17:20:12.593464073 +0000 UTC m=+3956.286735093" Nov 03 17:20:12.634055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634018 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634080 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634178 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634231 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634298 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634347 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634494 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634659 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.634751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634716 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q\") pod \"413cb794-f9b6-4a79-913d-6f0c14921c86\" (UID: \"413cb794-f9b6-4a79-913d-6f0c14921c86\") " Nov 03 17:20:12.635247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634339 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:12.635472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634593 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:12.635614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.634890 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:12.635785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.635692 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:12.636410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.636372 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.636586 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.636572 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413cb794-f9b6-4a79-913d-6f0c14921c86-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.636703 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.636688 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.636807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.636794 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.636957 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.636939 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:12.639483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.639224 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:12.639874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.639841 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:12.640378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.640343 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "kube-api-access-mnh5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:12.641022 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.641002 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token" pod "413cb794-f9b6-4a79-913d-6f0c14921c86" (UID: "413cb794-f9b6-4a79-913d-6f0c14921c86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:12.738631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.738593 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.738631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.738624 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnh5q\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-kube-api-access-mnh5q\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.738631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.738637 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413cb794-f9b6-4a79-913d-6f0c14921c86-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.738902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.738646 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.738902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.738655 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413cb794-f9b6-4a79-913d-6f0c14921c86-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:12.953129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.953095 4539 generic.go:356] "Generic (PLEG): container finished" podID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerID="7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca" exitCode=0 Nov 03 17:20:12.953664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.953146 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" Nov 03 17:20:12.953664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.953185 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" event={"ID":"413cb794-f9b6-4a79-913d-6f0c14921c86","Type":"ContainerDied","Data":"7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca"} Nov 03 17:20:12.953664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.953235 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4" event={"ID":"413cb794-f9b6-4a79-913d-6f0c14921c86","Type":"ContainerDied","Data":"8870e550f25a5564bcb439252957d00bcd57b2e7d199e66d8707d0a06d07a216"} Nov 03 17:20:12.953664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.953264 4539 scope.go:122] "RemoveContainer" containerID="7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca" Nov 03 17:20:12.968681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.968657 4539 scope.go:122] "RemoveContainer" containerID="7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca" Nov 03 17:20:12.968978 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:20:12.968953 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca\": container with ID starting with 7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca not found: ID does not exist" containerID="7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca" Nov 03 17:20:12.969052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:12.968986 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca"} err="failed to get container status \"7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca\": rpc error: code = NotFound desc = could not find container \"7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca\": container with ID starting with 7092b52d033abb0a0a6e21145bf5ed702fef3adae66bda5ea5a0e57e60eba3ca not found: ID does not exist" Nov 03 17:20:13.004586 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:13.004552 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4"] Nov 03 17:20:13.013175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:13.013140 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4nvf4"] Nov 03 17:20:13.182139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:13.182110 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:13.288604 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:13.288576 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v"] Nov 03 17:20:13.288794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:13.288766 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" containerID="cri-o://b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349" gracePeriod=30 Nov 03 17:20:14.281119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:14.281069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:14.470504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:14.470463 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" path="/var/lib/kubelet/pods/413cb794-f9b6-4a79-913d-6f0c14921c86/volumes" Nov 03 17:20:14.482167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:14.482139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:14.773353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:14.773326 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:15.728963 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:15.728931 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:16.366899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:16.366205 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:16.888399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:16.888369 4539 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 17:20:16.888859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:16.888443 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 17:20:16.995770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:16.995732 4539 generic.go:356] "Generic (PLEG): container finished" podID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerID="018f86b08eaf1d832c7b72cf32447a7e27fe4d3cf5f8c0833adadf6ebb9c5414" exitCode=0 Nov 03 17:20:16.995953 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:16.995812 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" event={"ID":"b1175722-921a-4307-9a59-cbbb8c6a48c9","Type":"ContainerDied","Data":"018f86b08eaf1d832c7b72cf32447a7e27fe4d3cf5f8c0833adadf6ebb9c5414"} Nov 03 17:20:17.149253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.149225 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:20:17.207900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.207863 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208108 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208009 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208120 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208179 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.208333 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208290 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208348 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208403 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208684 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208753 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.208874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.208795 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token\") pod \"b1175722-921a-4307-9a59-cbbb8c6a48c9\" (UID: \"b1175722-921a-4307-9a59-cbbb8c6a48c9\") " Nov 03 17:20:17.209485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.209374 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.209663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.209640 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.209784 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.209710 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:17.209861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.209778 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.210547 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.210574 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.210588 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.210601 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1175722-921a-4307-9a59-cbbb8c6a48c9-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.210615 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.212088 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "kube-api-access-7wp9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.213006 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.213218 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:17.217057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.215109 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy" pod "b1175722-921a-4307-9a59-cbbb8c6a48c9" (UID: "b1175722-921a-4307-9a59-cbbb8c6a48c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.311842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.311809 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.311842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.311835 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.311842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.311846 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1175722-921a-4307-9a59-cbbb8c6a48c9-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.312104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.311856 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wp9j\" (UniqueName: \"kubernetes.io/projected/b1175722-921a-4307-9a59-cbbb8c6a48c9-kube-api-access-7wp9j\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.775283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.775238 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:20:17.775513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.775307 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:20:17.959084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:17.959046 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:20:18.052508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.050744 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:18.072449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.070091 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" event={"ID":"b1175722-921a-4307-9a59-cbbb8c6a48c9","Type":"ContainerDied","Data":"2f706f7e59a2a3a68c969e6704b2ab4de858e34874d4c0842a8834224a199468"} Nov 03 17:20:18.072449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.070163 4539 scope.go:122] "RemoveContainer" containerID="018f86b08eaf1d832c7b72cf32447a7e27fe4d3cf5f8c0833adadf6ebb9c5414" Nov 03 17:20:18.072449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.070384 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7" Nov 03 17:20:18.148790 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.148761 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7"] Nov 03 17:20:18.153366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.153339 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-phqd7"] Nov 03 17:20:18.472488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.472462 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" path="/var/lib/kubelet/pods/b1175722-921a-4307-9a59-cbbb8c6a48c9/volumes" Nov 03 17:20:18.611234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.611203 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:20:18.655666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.655631 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.655877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.655696 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.655877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.655822 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.655877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.655862 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.656048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.655955 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.656048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.655962 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:18.656174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656117 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.656174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656160 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.656297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656201 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.656356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656312 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket\") pod \"1f6ca69b-7f63-4326-b93c-622577097b42\" (UID: \"1f6ca69b-7f63-4326-b93c-622577097b42\") " Nov 03 17:20:18.656499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656411 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:18.656574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656492 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:18.656692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656648 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:18.656977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.656773 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:18.657882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.657854 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.657882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.657883 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.658067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.657895 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6ca69b-7f63-4326-b93c-622577097b42-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.658067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.657908 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.658067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.657921 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.658965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.658931 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:18.658965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.658938 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:18.659259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.659239 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:18.659723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.659698 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64" pod "1f6ca69b-7f63-4326-b93c-622577097b42" (UID: "1f6ca69b-7f63-4326-b93c-622577097b42"). InnerVolumeSpecName "kube-api-access-52g64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:18.759505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.759417 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6ca69b-7f63-4326-b93c-622577097b42-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.759505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.759502 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52g64\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-kube-api-access-52g64\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.759505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.759515 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6ca69b-7f63-4326-b93c-622577097b42-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:18.759505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:18.759523 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6ca69b-7f63-4326-b93c-622577097b42-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:20:19.078632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.078597 4539 generic.go:356] "Generic (PLEG): container finished" podID="1f6ca69b-7f63-4326-b93c-622577097b42" containerID="b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349" exitCode=0 Nov 03 17:20:19.079147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.078653 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" Nov 03 17:20:19.079147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.078681 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" event={"ID":"1f6ca69b-7f63-4326-b93c-622577097b42","Type":"ContainerDied","Data":"b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349"} Nov 03 17:20:19.079147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.078738 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v" event={"ID":"1f6ca69b-7f63-4326-b93c-622577097b42","Type":"ContainerDied","Data":"e0c8caaa67b33885b7ed0f24682a68ce4dc7d69783444f6140d0634f9eff75a2"} Nov 03 17:20:19.079147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.078766 4539 scope.go:122] "RemoveContainer" containerID="b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349" Nov 03 17:20:19.090085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.090065 4539 scope.go:122] "RemoveContainer" containerID="b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349" Nov 03 17:20:19.090389 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:20:19.090368 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349\": container with ID starting with b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349 not found: ID does not exist" containerID="b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349" Nov 03 17:20:19.090540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.090398 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349"} err="failed to get container status \"b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349\": rpc error: code = NotFound desc = could not find container \"b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349\": container with ID starting with b1d79afc0e134ca3e10d1b46a03a91ae6ce244b732749580c87c32b14d368349 not found: ID does not exist" Nov 03 17:20:19.144678 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.132490 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v"] Nov 03 17:20:19.148997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.148961 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-x8q9v"] Nov 03 17:20:19.600295 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:19.586006 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:20.472823 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:20.472792 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" path="/var/lib/kubelet/pods/1f6ca69b-7f63-4326-b93c-622577097b42/volumes" Nov 03 17:20:21.216124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:21.216084 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:22.916058 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:22.916020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:24.498448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:24.489634 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:26.019450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:26.016288 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:26.161094 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:20:26.161058 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:20:27.703182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:27.703125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:29.290051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:29.286486 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:30.801353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:30.801307 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:32.499186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:32.499150 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:33.983459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:33.983400 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:35.586200 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:35.586156 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:37.252374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:37.252264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:38.873458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:38.868744 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:40.482673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:40.482639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:42.079562 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:42.074045 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:43.772031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:43.771974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:45.317116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:45.316022 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:46.988000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:46.987968 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:48.567551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:48.567469 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:50.097498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:50.097461 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:51.769348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:51.769313 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:53.320620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:53.320584 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:54.901450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:54.891511 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:56.159649 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:20:56.159615 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:20:56.493602 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:56.483569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:58.073653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:58.073619 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:20:59.716058 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:20:59.716019 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:01.764460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:01.764405 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:03.425032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:03.424982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:04.968987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:04.968948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:06.569081 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:06.569029 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:08.178017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:08.177988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:09.043276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.043246 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw"] Nov 03 17:21:09.044040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044022 4539 cpu_manager.go:414] "containerMap: removing container" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" Nov 03 17:21:09.044040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044041 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044057 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044063 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044077 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044082 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044190 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="413cb794-f9b6-4a79-913d-6f0c14921c86" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044199 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6ca69b-7f63-4326-b93c-622577097b42" containerName="istio-proxy" Nov 03 17:21:09.044234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.044207 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1175722-921a-4307-9a59-cbbb8c6a48c9" containerName="istio-proxy" Nov 03 17:21:09.048103 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.048080 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.077057 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.077035 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89df9a-openshift-default-dockercfg-hp8vk\"" Nov 03 17:21:09.092648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.092620 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw"] Nov 03 17:21:09.218136 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218090 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218208 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjpkb\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218313 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218507 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218556 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218687 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218750 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218849 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.218911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.218889 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.320925 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.320831 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.320925 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.320907 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.320944 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.320996 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321044 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjpkb\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321111 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321243 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321283 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321342 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321362 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321407 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321754 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.321962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.321910 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.322373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.322347 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.323947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.323927 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.324122 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.324106 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.342797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.342774 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjpkb\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.344162 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.344141 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.364084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.364062 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:09.771758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.771652 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw"] Nov 03 17:21:09.795107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.793133 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:21:09.795107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.793222 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:21:09.795107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.793261 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:21:09.883801 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:09.883765 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:10.580257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:10.580216 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" event={"ID":"103edeb3-8c23-4d58-a70c-bb4daa474873","Type":"ContainerStarted","Data":"d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526"} Nov 03 17:21:10.580699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:10.580278 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" event={"ID":"103edeb3-8c23-4d58-a70c-bb4daa474873","Type":"ContainerStarted","Data":"60b3ee06768ba6bb011bdac7ce506e6f81d6d202abc538a90a568185bbf78966"} Nov 03 17:21:10.635299 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:10.635253 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podStartSLOduration=1.635233832 podStartE2EDuration="1.635233832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:10.632565678 +0000 UTC m=+4014.325836699" watchObservedRunningTime="2026-11-03 17:21:10.635233832 +0000 UTC m=+4014.328504853" Nov 03 17:21:11.364718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:11.364684 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:11.366811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:11.366318 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 17:21:11.366811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:11.366370 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 17:21:11.928465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:11.923880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:12.249502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.244631 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84"] Nov 03 17:21:12.257483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.254307 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.259521 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.259154 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84"] Nov 03 17:21:12.278515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.278486 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-472bee\"/\"test-model-472bee-kserve-self-signed-certs\"" Nov 03 17:21:12.278860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.278840 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-472bee\"/\"default-dockercfg-q9hzv\"" Nov 03 17:21:12.287285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.287261 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.287655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.287633 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.288008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.287970 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.288229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.288211 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76w5\" (UniqueName: \"kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.288437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.288403 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.288624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.288591 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.316648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.315688 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-472bee\"/\"kube-root-ca.crt\"" Nov 03 17:21:12.316648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.315805 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-472bee\"/\"openshift-service-ca.crt\"" Nov 03 17:21:12.357822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.357652 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:12.367005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.364851 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 17:21:12.367005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.364906 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 17:21:12.390174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390147 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390186 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76w5\" (UniqueName: \"kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390241 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390300 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390328 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390617 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390373 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390671 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390638 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390679 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.390795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.390774 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.392941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.392910 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.393487 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.393467 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.463885 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.459395 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76w5\" (UniqueName: \"kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5\") pod \"test-model-472bee-kserve-578d88976f-mkh84\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:12.592453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:12.592030 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:13.152847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.152776 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84"] Nov 03 17:21:13.364649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.364615 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 17:21:13.364838 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.364670 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 17:21:13.781331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.781299 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" event={"ID":"144603bc-fbc4-4006-91ff-7f08ab5d1769","Type":"ContainerStarted","Data":"dffdb10be8f1cfdcfdbf226533a76d8a5e84a33f2d2e2a69778598b76c8d9a5b"} Nov 03 17:21:13.781554 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.781341 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" event={"ID":"144603bc-fbc4-4006-91ff-7f08ab5d1769","Type":"ContainerStarted","Data":"d535a839ddf4c2de0d8c6bce9d64bf503975e9b8d3a252c04386e894b92baad3"} Nov 03 17:21:13.876455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.876286 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:13.895448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:13.893328 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" podStartSLOduration=1.893307104 podStartE2EDuration="1.893307104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:13.857822872 +0000 UTC m=+4017.551093893" watchObservedRunningTime="2026-11-03 17:21:13.893307104 +0000 UTC m=+4017.586578127" Nov 03 17:21:14.364682 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:14.364647 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 17:21:14.365101 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:14.364700 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 17:21:14.606104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:14.606070 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:15.365496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:15.364857 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 17:21:15.365496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:15.364931 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 17:21:15.368714 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:15.368683 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:15.528388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:15.528351 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:15.949096 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:15.949059 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:16.364726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:16.364688 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 17:21:16.364935 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:16.364749 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 17:21:17.290494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:17.286339 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:17.370107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:17.370080 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:17.370510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:17.370482 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:17.371100 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:17.371077 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:21:18.859821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:18.859791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:20.646069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:20.646038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:22.277756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:22.277715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:22.606545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:22.606398 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:22.619870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:22.619835 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:22.877730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:22.877641 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:21:23.896066 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:23.896018 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:25.484393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:25.484361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:26.158418 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:21:26.158385 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:21:27.156898 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:27.156865 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:28.728451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:28.723996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:30.413009 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:30.412973 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:31.984823 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:31.984790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:33.621634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:33.621541 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:35.298082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:35.298041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:36.961951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:36.961909 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:38.505594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:38.505560 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:40.100388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:40.100351 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:41.613918 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:41.613882 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:44.323011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:44.322961 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:45.125307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:45.125275 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:46.915196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:46.915130 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:48.819157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:48.819121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:50.440379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:50.440341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:52.069163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:52.069118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:53.914713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:53.913087 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:54.182292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.182209 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc"] Nov 03 17:21:54.188102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.188080 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.212840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.212808 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc"] Nov 03 17:21:54.343131 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343067 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343140 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdp8p\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343257 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343295 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343374 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343446 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343634 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343677 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.343808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.343755 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445405 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445549 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445609 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445659 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445701 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdp8p\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445755 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445780 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.445807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445801 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.446135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445827 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.446135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445855 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.446135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.445986 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.446135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.446041 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.446373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.446289 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.447172 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.447143 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.448503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.448479 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.448799 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.448779 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.468996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.468958 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.469655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.469625 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdp8p\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.503906 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.503874 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:54.831555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.831525 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc"] Nov 03 17:21:54.831707 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:21:54.831617 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778356c7_9507_4aca_91ed_8ed015d90ae4.slice/crio-4e265c49d42853b1dcaeedfc332eacb93970b7b97cdc04f9fc72eac1d2441e04 WatchSource:0}: Error finding container 4e265c49d42853b1dcaeedfc332eacb93970b7b97cdc04f9fc72eac1d2441e04: Status 404 returned error can't find the container with id 4e265c49d42853b1dcaeedfc332eacb93970b7b97cdc04f9fc72eac1d2441e04 Nov 03 17:21:54.835524 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.835491 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:21:54.835626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.835559 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:21:54.835626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:54.835591 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:21:55.178285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.178191 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" event={"ID":"778356c7-9507-4aca-91ed-8ed015d90ae4","Type":"ContainerStarted","Data":"16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f"} Nov 03 17:21:55.178887 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.178856 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" event={"ID":"778356c7-9507-4aca-91ed-8ed015d90ae4","Type":"ContainerStarted","Data":"4e265c49d42853b1dcaeedfc332eacb93970b7b97cdc04f9fc72eac1d2441e04"} Nov 03 17:21:55.225720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.225629 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podStartSLOduration=1.22561003 podStartE2EDuration="1.22561003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:55.225419345 +0000 UTC m=+4058.918690382" watchObservedRunningTime="2026-11-03 17:21:55.22561003 +0000 UTC m=+4058.918881051" Nov 03 17:21:55.516482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.512354 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:21:55.520484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.519570 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 17:21:55.520484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.519636 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 17:21:55.760494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:55.760451 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:56.159118 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:21:56.159065 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:21:56.513506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:56.509797 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 17:21:56.513506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:56.509855 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 17:21:57.324456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:57.323613 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:57.505374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:57.504837 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 17:21:57.505374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:57.504917 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 17:21:58.514450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:58.510571 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 17:21:58.514450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:58.510670 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 17:21:58.903449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:58.903337 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:21:59.522451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:59.508611 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 17:21:59.522451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:21:59.508684 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 17:22:00.516452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:00.507167 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 03 17:22:00.516452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:00.507239 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 03 17:22:00.612480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:00.611979 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:01.511779 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:01.511736 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:22:01.512262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:01.512233 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:22:01.512977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:01.512951 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:22:02.161351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:02.161200 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:03.800695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:03.800653 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:05.390365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:05.390279 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:06.915910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:06.915872 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:08.636497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:08.632870 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:10.178450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:10.178019 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:11.752537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:11.752118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:13.470334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:13.470304 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:15.160370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:15.160329 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:16.084132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:16.084093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:16.954102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:16.954066 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:17.996140 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:17.996108 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn"] Nov 03 17:22:17.996702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:17.996413 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" containerID="cri-o://f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c" gracePeriod=30 Nov 03 17:22:18.937644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:18.931746 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:20.557221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:20.557192 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:22.168749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:22.168709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:23.297796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.297771 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:22:23.465603 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.465567 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.467385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.467362 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.468114 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468081 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.468291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.467234 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:23.468460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468391 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.468853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468702 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.468853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468807 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.468853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468838 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.469184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468920 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.469184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.468963 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert\") pod \"11212458-38d0-48bd-a512-ecb6b9b24dc1\" (UID: \"11212458-38d0-48bd-a512-ecb6b9b24dc1\") " Nov 03 17:22:23.471251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.469583 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:23.471251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.469905 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:23.471251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.470169 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:23.471251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.470728 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472165 4539 generic.go:356] "Generic (PLEG): container finished" podID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerID="f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c" exitCode=0 Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472523 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" event={"ID":"11212458-38d0-48bd-a512-ecb6b9b24dc1","Type":"ContainerDied","Data":"f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c"} Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472591 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" event={"ID":"11212458-38d0-48bd-a512-ecb6b9b24dc1","Type":"ContainerDied","Data":"3d15a7c76a33af0e3bf07f8b0d38c04504faa6fcc52067ef703645082d80377e"} Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472626 4539 scope.go:122] "RemoveContainer" containerID="f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472812 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472911 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.472994 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.473008 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.473017 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.473040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.473025 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11212458-38d0-48bd-a512-ecb6b9b24dc1-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.476164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.476128 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "kube-api-access-cgdh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:23.477119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.477086 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:23.477680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.477655 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:23.478816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.478792 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy" pod "11212458-38d0-48bd-a512-ecb6b9b24dc1" (UID: "11212458-38d0-48bd-a512-ecb6b9b24dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:23.509324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.509297 4539 scope.go:122] "RemoveContainer" containerID="f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c" Nov 03 17:22:23.509815 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:22:23.509783 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c\": container with ID starting with f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c not found: ID does not exist" containerID="f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c" Nov 03 17:22:23.509923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.509825 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c"} err="failed to get container status \"f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c\": rpc error: code = NotFound desc = could not find container \"f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c\": container with ID starting with f8171f9f965f5591c5f0b9a18182992579a9c2eb6cfa87ece1f77cabe0ced28c not found: ID does not exist" Nov 03 17:22:23.574594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.574510 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.574594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.574549 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdh7\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-kube-api-access-cgdh7\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.574594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.574566 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.574594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.574581 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11212458-38d0-48bd-a512-ecb6b9b24dc1-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:23.744098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.744051 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:23.836900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.836796 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn"] Nov 03 17:22:23.841647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:23.841614 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-tbxvn"] Nov 03 17:22:24.470550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:24.470509 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" path="/var/lib/kubelet/pods/11212458-38d0-48bd-a512-ecb6b9b24dc1/volumes" Nov 03 17:22:25.375611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:25.375579 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:26.162002 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:22:26.161959 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:22:27.103306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:27.100068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:28.815274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:28.815178 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:30.305698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:30.305660 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:31.746871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:31.746833 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz"] Nov 03 17:22:31.747408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:31.747075 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" containerID="cri-o://ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac" gracePeriod=30 Nov 03 17:22:32.016514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:32.012976 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:33.679464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:33.679392 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:35.212447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:35.209452 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:36.451842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:36.451804 4539 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:22:36.452288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:36.451866 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:22:36.789106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:36.789069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:37.043275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.043241 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:22:37.210654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210620 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.210846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210692 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.210846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210777 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.210846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210797 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.211036 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210870 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.211036 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210904 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.211036 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210942 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:37.211036 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.210956 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.211336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211124 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.211336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211162 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:37.211336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211203 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs\") pod \"24266884-b424-4358-9359-594731ec8628\" (UID: \"24266884-b424-4358-9359-594731ec8628\") " Nov 03 17:22:37.211336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211226 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:37.211576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211552 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:37.211821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211793 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:37.211996 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.211979 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.212069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.212001 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.212069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.212016 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.212069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.212029 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24266884-b424-4358-9359-594731ec8628-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.212069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.212040 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.213945 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.213914 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:37.214236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.214212 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "kube-api-access-hxpr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:37.214329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.214251 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:37.214440 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.214402 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo" pod "24266884-b424-4358-9359-594731ec8628" (UID: "24266884-b424-4358-9359-594731ec8628"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:37.313118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.313080 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24266884-b424-4358-9359-594731ec8628-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.313118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.313117 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.313118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.313126 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24266884-b424-4358-9359-594731ec8628-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.313118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.313135 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxpr7\" (UniqueName: \"kubernetes.io/projected/24266884-b424-4358-9359-594731ec8628-kube-api-access-hxpr7\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:37.597396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.597302 4539 generic.go:356] "Generic (PLEG): container finished" podID="24266884-b424-4358-9359-594731ec8628" containerID="ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac" exitCode=0 Nov 03 17:22:37.597396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.597366 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" Nov 03 17:22:37.597902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.597392 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" event={"ID":"24266884-b424-4358-9359-594731ec8628","Type":"ContainerDied","Data":"ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac"} Nov 03 17:22:37.597902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.597470 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz" event={"ID":"24266884-b424-4358-9359-594731ec8628","Type":"ContainerDied","Data":"e6c4f42a5cd9e1c990ba66c9d88462a3ebdb72ae0b35a6ba5b2ad777ff7a4a3c"} Nov 03 17:22:37.597902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.597495 4539 scope.go:122] "RemoveContainer" containerID="ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac" Nov 03 17:22:37.605704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.605669 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k"] Nov 03 17:22:37.605931 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.605906 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" containerID="cri-o://e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f" gracePeriod=30 Nov 03 17:22:37.612667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.612641 4539 scope.go:122] "RemoveContainer" containerID="ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac" Nov 03 17:22:37.613012 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:22:37.612992 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac\": container with ID starting with ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac not found: ID does not exist" containerID="ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac" Nov 03 17:22:37.613091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.613021 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac"} err="failed to get container status \"ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac\": rpc error: code = NotFound desc = could not find container \"ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac\": container with ID starting with ebaffd65d04c5edc37e86ea644251fb7ae2ca95e6aa2948042f501f3a3a771ac not found: ID does not exist" Nov 03 17:22:37.660235 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.660160 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz"] Nov 03 17:22:37.665836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:37.665805 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-rqtdz"] Nov 03 17:22:38.096357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:38.096318 4539 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:22:38.096590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:38.096395 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:22:38.406162 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:38.406069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:38.478485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:38.478453 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24266884-b424-4358-9359-594731ec8628" path="/var/lib/kubelet/pods/24266884-b424-4358-9359-594731ec8628/volumes" Nov 03 17:22:39.952138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:39.952049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:41.605452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:41.582514 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:42.904461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:42.904414 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 17:22:43.001640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001604 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.001851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001676 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.001851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001714 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.001851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001763 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.001851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001793 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.001851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001814 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.001851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001842 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.002254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001864 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.002254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.001918 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data\") pod \"e56e708e-f97c-44b7-97bd-c31970eab99c\" (UID: \"e56e708e-f97c-44b7-97bd-c31970eab99c\") " Nov 03 17:22:43.002254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.002046 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:43.002456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.002357 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:43.002781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.002758 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e56e708e-f97c-44b7-97bd-c31970eab99c-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.002781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.002785 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.002949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.002811 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:43.003012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.002993 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:43.003277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.003252 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:43.005030 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.005001 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:43.005236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.005214 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:43.005326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.005265 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "kube-api-access-4rrwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:43.007555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.007530 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy" pod "e56e708e-f97c-44b7-97bd-c31970eab99c" (UID: "e56e708e-f97c-44b7-97bd-c31970eab99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:43.104310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104269 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rrwq\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-kube-api-access-4rrwq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.104310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104308 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.104594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104322 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.104594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104336 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.104594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104348 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.104594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104361 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.104594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.104373 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e56e708e-f97c-44b7-97bd-c31970eab99c-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:22:43.310324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.310175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:43.659407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.659318 4539 generic.go:356] "Generic (PLEG): container finished" podID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerID="e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f" exitCode=0 Nov 03 17:22:43.659407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.659362 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" Nov 03 17:22:43.659644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.659404 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" event={"ID":"e56e708e-f97c-44b7-97bd-c31970eab99c","Type":"ContainerDied","Data":"e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f"} Nov 03 17:22:43.659644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.659471 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k" event={"ID":"e56e708e-f97c-44b7-97bd-c31970eab99c","Type":"ContainerDied","Data":"598806f6110d560f6a7fbc962617d836144731edb2d7de7e90e601e92378873d"} Nov 03 17:22:43.659644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.659493 4539 scope.go:122] "RemoveContainer" containerID="e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f" Nov 03 17:22:43.673784 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.673768 4539 scope.go:122] "RemoveContainer" containerID="e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f" Nov 03 17:22:43.674075 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:22:43.674042 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f\": container with ID starting with e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f not found: ID does not exist" containerID="e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f" Nov 03 17:22:43.674170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.674081 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f"} err="failed to get container status \"e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f\": rpc error: code = NotFound desc = could not find container \"e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f\": container with ID starting with e85a53105d1d01f4ab604f476f6585da4c6298f49ee954b537d27c06ccbb292f not found: ID does not exist" Nov 03 17:22:43.724731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.724690 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k"] Nov 03 17:22:43.734912 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:43.734877 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-nxv5k"] Nov 03 17:22:44.472292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:44.472251 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" path="/var/lib/kubelet/pods/e56e708e-f97c-44b7-97bd-c31970eab99c/volumes" Nov 03 17:22:44.925697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:44.921771 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:46.496538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:46.496494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:48.108148 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:48.108114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:49.685901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:49.685864 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:51.264272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:51.263235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:52.920450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:52.915111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:54.560561 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:54.560531 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:56.091613 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:56.091579 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:56.179536 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:22:56.179496 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:22:57.697858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:57.695545 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:22:59.248709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:22:59.248657 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:00.737694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:00.737655 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:02.406477 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:02.406363 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:04.425616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:04.425579 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:06.079895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:06.079859 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:07.670098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:07.670055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:09.274178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:09.274132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:10.787930 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:10.787898 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:12.384340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:12.384284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:13.919451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:13.917476 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:15.461032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:15.460995 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:16.210024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:16.209988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:17.024452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:17.019113 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:18.590193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:18.590145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:20.290192 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:20.290132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:21.842587 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:21.842527 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:23.421353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:23.421315 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:24.961060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:24.961020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:26.159789 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:23:26.159749 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:23:26.613516 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:26.613468 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:28.160515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:28.160477 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:29.782051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:29.782011 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:31.368938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:31.368904 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:32.882503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:32.882468 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:34.514507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:34.514472 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:36.133808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:36.133769 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:37.680983 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:37.680940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:39.176329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:39.176293 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc"] Nov 03 17:23:39.240296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:39.240250 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:39.410456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:39.409311 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:39.817658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:39.817616 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg"] Nov 03 17:23:39.894894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:39.894857 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:41.167277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:41.167223 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:42.713454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:42.710174 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:44.279142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:44.279109 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:45.799848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:45.799810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:47.207233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:47.207194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:47.371562 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:47.371515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:47.542492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:47.542450 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:49.201614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:49.201578 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:50.770961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:50.770924 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:52.309527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:52.309179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:53.895411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:53.895378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:54.231621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.231532 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" podUID="0dd9deaa-49be-411d-9b7d-7d9722f9918f" containerName="main" containerID="cri-o://f0b81fd623adf1570b85c6e42ad359b863e403950dbdb26b1c067d71eeee9fce" gracePeriod=45 Nov 03 17:23:54.366606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.366572 4539 generic.go:356] "Generic (PLEG): container finished" podID="0dd9deaa-49be-411d-9b7d-7d9722f9918f" containerID="f0b81fd623adf1570b85c6e42ad359b863e403950dbdb26b1c067d71eeee9fce" exitCode=0 Nov 03 17:23:54.366774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.366642 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" event={"ID":"0dd9deaa-49be-411d-9b7d-7d9722f9918f","Type":"ContainerDied","Data":"f0b81fd623adf1570b85c6e42ad359b863e403950dbdb26b1c067d71eeee9fce"} Nov 03 17:23:54.560973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.560944 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:23:54.733310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733268 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home\") pod \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " Nov 03 17:23:54.733607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733590 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir\") pod \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " Nov 03 17:23:54.733687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733672 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache\") pod \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " Nov 03 17:23:54.733749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733714 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p\" (UniqueName: \"kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p\") pod \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " Nov 03 17:23:54.733749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733591 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home" pod "0dd9deaa-49be-411d-9b7d-7d9722f9918f" (UID: "0dd9deaa-49be-411d-9b7d-7d9722f9918f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:54.733855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733747 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs\") pod \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " Nov 03 17:23:54.733855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733712 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir" pod "0dd9deaa-49be-411d-9b7d-7d9722f9918f" (UID: "0dd9deaa-49be-411d-9b7d-7d9722f9918f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:54.733855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733839 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm\") pod \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\" (UID: \"0dd9deaa-49be-411d-9b7d-7d9722f9918f\") " Nov 03 17:23:54.733855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.733829 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache" pod "0dd9deaa-49be-411d-9b7d-7d9722f9918f" (UID: "0dd9deaa-49be-411d-9b7d-7d9722f9918f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:54.734548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.734489 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:54.734548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.734512 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:54.734548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.734527 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:54.736327 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.736255 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm" pod "0dd9deaa-49be-411d-9b7d-7d9722f9918f" (UID: "0dd9deaa-49be-411d-9b7d-7d9722f9918f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:54.736465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.736327 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p" pod "0dd9deaa-49be-411d-9b7d-7d9722f9918f" (UID: "0dd9deaa-49be-411d-9b7d-7d9722f9918f"). InnerVolumeSpecName "kube-api-access-88l5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:54.736465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.736349 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs" pod "0dd9deaa-49be-411d-9b7d-7d9722f9918f" (UID: "0dd9deaa-49be-411d-9b7d-7d9722f9918f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:54.835923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.835885 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88l5p\" (UniqueName: \"kubernetes.io/projected/0dd9deaa-49be-411d-9b7d-7d9722f9918f-kube-api-access-88l5p\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:54.835923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.835914 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dd9deaa-49be-411d-9b7d-7d9722f9918f-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:54.835923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.835923 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dd9deaa-49be-411d-9b7d-7d9722f9918f-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:54.864854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:54.864816 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" podUID="f2309506-25e4-4d6b-a718-11a3a5991386" containerName="main" containerID="cri-o://883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5" gracePeriod=45 Nov 03 17:23:55.159867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.159841 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:23:55.256531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.256502 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir\") pod \"f2309506-25e4-4d6b-a718-11a3a5991386\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " Nov 03 17:23:55.256668 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.256553 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home\") pod \"f2309506-25e4-4d6b-a718-11a3a5991386\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " Nov 03 17:23:55.256668 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.256602 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache\") pod \"f2309506-25e4-4d6b-a718-11a3a5991386\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " Nov 03 17:23:55.256808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.256796 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm\") pod \"f2309506-25e4-4d6b-a718-11a3a5991386\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " Nov 03 17:23:55.256869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.256859 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs\") pod \"f2309506-25e4-4d6b-a718-11a3a5991386\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " Nov 03 17:23:55.256944 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.256932 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf\" (UniqueName: \"kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf\") pod \"f2309506-25e4-4d6b-a718-11a3a5991386\" (UID: \"f2309506-25e4-4d6b-a718-11a3a5991386\") " Nov 03 17:23:55.257966 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.257932 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache" pod "f2309506-25e4-4d6b-a718-11a3a5991386" (UID: "f2309506-25e4-4d6b-a718-11a3a5991386"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:55.258169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.258149 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir" pod "f2309506-25e4-4d6b-a718-11a3a5991386" (UID: "f2309506-25e4-4d6b-a718-11a3a5991386"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:55.258350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.258331 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home" pod "f2309506-25e4-4d6b-a718-11a3a5991386" (UID: "f2309506-25e4-4d6b-a718-11a3a5991386"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:55.260508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.260485 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf" pod "f2309506-25e4-4d6b-a718-11a3a5991386" (UID: "f2309506-25e4-4d6b-a718-11a3a5991386"). InnerVolumeSpecName "kube-api-access-p9dpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:55.260704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.260679 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm" pod "f2309506-25e4-4d6b-a718-11a3a5991386" (UID: "f2309506-25e4-4d6b-a718-11a3a5991386"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:55.261346 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.261320 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs" pod "f2309506-25e4-4d6b-a718-11a3a5991386" (UID: "f2309506-25e4-4d6b-a718-11a3a5991386"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:55.358459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.358387 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:55.358459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.358436 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:55.358459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.358447 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:55.358459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.358454 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2309506-25e4-4d6b-a718-11a3a5991386-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:55.358459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.358462 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2309506-25e4-4d6b-a718-11a3a5991386-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:55.358459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.358470 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9dpf\" (UniqueName: \"kubernetes.io/projected/f2309506-25e4-4d6b-a718-11a3a5991386-kube-api-access-p9dpf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:23:55.376406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.376362 4539 generic.go:356] "Generic (PLEG): container finished" podID="f2309506-25e4-4d6b-a718-11a3a5991386" containerID="883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5" exitCode=0 Nov 03 17:23:55.376624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.376418 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" Nov 03 17:23:55.376624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.376438 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" event={"ID":"f2309506-25e4-4d6b-a718-11a3a5991386","Type":"ContainerDied","Data":"883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5"} Nov 03 17:23:55.376624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.376480 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg" event={"ID":"f2309506-25e4-4d6b-a718-11a3a5991386","Type":"ContainerDied","Data":"b5cfa19b8bc6f92a91fb4741135dc385057031772d90d3a2dbbb2bc7ec4c9069"} Nov 03 17:23:55.376624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.376512 4539 scope.go:122] "RemoveContainer" containerID="883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5" Nov 03 17:23:55.378228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.378202 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" event={"ID":"0dd9deaa-49be-411d-9b7d-7d9722f9918f","Type":"ContainerDied","Data":"b6ae971a5d3cc45c14f53e460c0977f253638523d2b9dce0bd9f711322a54361"} Nov 03 17:23:55.378391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.378326 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc" Nov 03 17:23:55.389467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.389407 4539 scope.go:122] "RemoveContainer" containerID="883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5" Nov 03 17:23:55.389729 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:23:55.389708 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5\": container with ID starting with 883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5 not found: ID does not exist" containerID="883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5" Nov 03 17:23:55.389802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.389736 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5"} err="failed to get container status \"883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5\": rpc error: code = NotFound desc = could not find container \"883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5\": container with ID starting with 883b408760ed6f775a09b32bae2688a8831d548221f84fc049ee0bcb238b01e5 not found: ID does not exist" Nov 03 17:23:55.389802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.389751 4539 scope.go:122] "RemoveContainer" containerID="f0b81fd623adf1570b85c6e42ad359b863e403950dbdb26b1c067d71eeee9fce" Nov 03 17:23:55.427411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.427342 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg"] Nov 03 17:23:55.433978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.433952 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-24cb6d/test-model-24cb6d-kserve-f9456769c-j7fqg"] Nov 03 17:23:55.459320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.459289 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc"] Nov 03 17:23:55.464412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.464377 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fc40cc/test-model-fc40cc-kserve-8458d96ff7-fnfzc"] Nov 03 17:23:55.561075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.561022 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:55.957031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:55.956989 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:56.174539 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:23:56.174438 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:23:56.342018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:56.341987 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:56.478187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:56.478146 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd9deaa-49be-411d-9b7d-7d9722f9918f" path="/var/lib/kubelet/pods/0dd9deaa-49be-411d-9b7d-7d9722f9918f/volumes" Nov 03 17:23:56.478928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:56.478891 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2309506-25e4-4d6b-a718-11a3a5991386" path="/var/lib/kubelet/pods/f2309506-25e4-4d6b-a718-11a3a5991386/volumes" Nov 03 17:23:57.289811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:57.289777 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:23:59.107964 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:23:59.107865 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:00.749773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:00.749738 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:01.644982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:01.644948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:01.848585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:01.848518 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:02.717445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:02.710533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:03.356524 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:03.356494 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:04.287389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:04.287352 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq"] Nov 03 17:24:04.377364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:04.377331 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:04.557305 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:04.557068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:04.855215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:04.855083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:06.080482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:06.079811 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:07.685970 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:07.685929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:09.340635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:09.340597 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:09.517892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:09.517856 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:10.442413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:10.442378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:10.603637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:10.603604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:11.402847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:11.402810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:11.758231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:11.758137 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:13.001169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:13.001051 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:14.553451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:14.539003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:16.090067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:16.090013 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:16.380645 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:16.380532 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:17.311977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:17.311938 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:24:17.330722 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:17.330700 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:24:17.703612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:17.703516 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:17.898085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:17.898050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:19.325212 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:19.325104 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" podUID="9da67c33-2e77-4f6b-8897-ab346420a222" containerName="main" containerID="cri-o://97dc3d96e9fba8b11b16c1982b10dac17a2549853ad9444a56c7902aad4c0d15" gracePeriod=45 Nov 03 17:24:19.457455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:19.454603 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" podUID="9da67c33-2e77-4f6b-8897-ab346420a222" containerName="main" probeResult="failure" output="Get \"https://10.133.0.190:8000/ready\": dial tcp 10.133.0.190:8000: connect: connection refused" Nov 03 17:24:19.698403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:19.698312 4539 generic.go:356] "Generic (PLEG): container finished" podID="9da67c33-2e77-4f6b-8897-ab346420a222" containerID="97dc3d96e9fba8b11b16c1982b10dac17a2549853ad9444a56c7902aad4c0d15" exitCode=0 Nov 03 17:24:19.698696 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:19.698669 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" event={"ID":"9da67c33-2e77-4f6b-8897-ab346420a222","Type":"ContainerDied","Data":"97dc3d96e9fba8b11b16c1982b10dac17a2549853ad9444a56c7902aad4c0d15"} Nov 03 17:24:19.937491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:19.920918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:20.176895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.176818 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:20.233612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.233582 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:24:20.279828 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.279792 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm\") pod \"9da67c33-2e77-4f6b-8897-ab346420a222\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " Nov 03 17:24:20.280256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280229 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache\") pod \"9da67c33-2e77-4f6b-8897-ab346420a222\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " Nov 03 17:24:20.280400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280284 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk\" (UniqueName: \"kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk\") pod \"9da67c33-2e77-4f6b-8897-ab346420a222\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " Nov 03 17:24:20.280400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280311 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home\") pod \"9da67c33-2e77-4f6b-8897-ab346420a222\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " Nov 03 17:24:20.280542 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280409 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs\" (UniqueName: \"kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs\") pod \"9da67c33-2e77-4f6b-8897-ab346420a222\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " Nov 03 17:24:20.280542 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280519 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir\") pod \"9da67c33-2e77-4f6b-8897-ab346420a222\" (UID: \"9da67c33-2e77-4f6b-8897-ab346420a222\") " Nov 03 17:24:20.281229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280917 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home" pod "9da67c33-2e77-4f6b-8897-ab346420a222" (UID: "9da67c33-2e77-4f6b-8897-ab346420a222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:20.281229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.280987 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache" pod "9da67c33-2e77-4f6b-8897-ab346420a222" (UID: "9da67c33-2e77-4f6b-8897-ab346420a222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:20.281229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.281161 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:20.281229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.281173 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:20.281571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.281390 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir" pod "9da67c33-2e77-4f6b-8897-ab346420a222" (UID: "9da67c33-2e77-4f6b-8897-ab346420a222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:20.284415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.282991 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm" pod "9da67c33-2e77-4f6b-8897-ab346420a222" (UID: "9da67c33-2e77-4f6b-8897-ab346420a222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:20.284415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.283675 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk" pod "9da67c33-2e77-4f6b-8897-ab346420a222" (UID: "9da67c33-2e77-4f6b-8897-ab346420a222"). InnerVolumeSpecName "kube-api-access-ln6wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:20.285744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.285720 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs" pod "9da67c33-2e77-4f6b-8897-ab346420a222" (UID: "9da67c33-2e77-4f6b-8897-ab346420a222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:20.384464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.383741 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6wk\" (UniqueName: \"kubernetes.io/projected/9da67c33-2e77-4f6b-8897-ab346420a222-kube-api-access-ln6wk\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:20.384464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.383785 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9da67c33-2e77-4f6b-8897-ab346420a222-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:20.384464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.383826 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:20.384464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.383841 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9da67c33-2e77-4f6b-8897-ab346420a222-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:20.708271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.708242 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" Nov 03 17:24:20.708271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.708264 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq" event={"ID":"9da67c33-2e77-4f6b-8897-ab346420a222","Type":"ContainerDied","Data":"b97f3fe004b4b3e1fdd8284deb49949989ca6001d14c7b97caee3aeee08d7228"} Nov 03 17:24:20.708605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.708311 4539 scope.go:122] "RemoveContainer" containerID="97dc3d96e9fba8b11b16c1982b10dac17a2549853ad9444a56c7902aad4c0d15" Nov 03 17:24:20.769855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.769824 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq"] Nov 03 17:24:20.778660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:20.778624 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-527051/test-model-527051-kserve-75959dd887-sm9lq"] Nov 03 17:24:22.017444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:22.016981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:22.471608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:22.471575 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da67c33-2e77-4f6b-8897-ab346420a222" path="/var/lib/kubelet/pods/9da67c33-2e77-4f6b-8897-ab346420a222/volumes" Nov 03 17:24:23.768500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:23.768465 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:25.465342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:25.465308 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:26.161849 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:24:26.161781 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:24:26.847637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:26.847599 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:27.788374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:27.788340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:27.920265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:27.920214 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm"] Nov 03 17:24:27.920772 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:27.920484 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" podUID="1329b928-6285-4d41-8a3e-af0b55343c16" containerName="discovery" containerID="cri-o://174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842" gracePeriod=30 Nov 03 17:24:28.235571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.235535 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:24:28.317648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.317548 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.317648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.317625 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.317899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.317699 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.317899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.317778 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.318024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.317930 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.318075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.318034 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:28.318075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.318044 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.318188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.318142 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig\") pod \"1329b928-6285-4d41-8a3e-af0b55343c16\" (UID: \"1329b928-6285-4d41-8a3e-af0b55343c16\") " Nov 03 17:24:28.318890 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.318864 4539 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-ca-configmap\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.321413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.321369 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:28.321754 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.321713 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:28.323209 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.323184 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:28.323336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.323199 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:28.323382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.323361 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "kube-api-access-wjqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:28.323773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.323743 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token" pod "1329b928-6285-4d41-8a3e-af0b55343c16" (UID: "1329b928-6285-4d41-8a3e-af0b55343c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:28.420653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.420615 4539 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-cacerts\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.420653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.420656 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqkp\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-kube-api-access-wjqkp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.420653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.420670 4539 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1329b928-6285-4d41-8a3e-af0b55343c16-local-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.420977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.420685 4539 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-csr-dns-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.420977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.420695 4539 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1329b928-6285-4d41-8a3e-af0b55343c16-istio-kubeconfig\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.420977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.420703 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1329b928-6285-4d41-8a3e-af0b55343c16-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.778881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.778846 4539 generic.go:356] "Generic (PLEG): container finished" podID="1329b928-6285-4d41-8a3e-af0b55343c16" containerID="174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842" exitCode=0 Nov 03 17:24:28.779076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.778906 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" Nov 03 17:24:28.779076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.778936 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" event={"ID":"1329b928-6285-4d41-8a3e-af0b55343c16","Type":"ContainerDied","Data":"174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842"} Nov 03 17:24:28.779076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.778994 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm" event={"ID":"1329b928-6285-4d41-8a3e-af0b55343c16","Type":"ContainerDied","Data":"9cd588944c7b26ee64cb8535ae9868aef5191a6b18aaed9e3e54b3d358934300"} Nov 03 17:24:28.779076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.779014 4539 scope.go:122] "RemoveContainer" containerID="174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842" Nov 03 17:24:28.793916 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.793893 4539 scope.go:122] "RemoveContainer" containerID="174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842" Nov 03 17:24:28.794194 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:24:28.794172 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842\": container with ID starting with 174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842 not found: ID does not exist" containerID="174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842" Nov 03 17:24:28.794288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.794203 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842"} err="failed to get container status \"174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842\": rpc error: code = NotFound desc = could not find container \"174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842\": container with ID starting with 174b644249389ae616ef02e386fa261cecb0e9b2074b192bf10c28caf224c842 not found: ID does not exist" Nov 03 17:24:28.836619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.836589 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm"] Nov 03 17:24:28.847921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:28.847890 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfwwm"] Nov 03 17:24:29.546328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:29.546287 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:30.476135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:30.476100 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1329b928-6285-4d41-8a3e-af0b55343c16" path="/var/lib/kubelet/pods/1329b928-6285-4d41-8a3e-af0b55343c16/volumes" Nov 03 17:24:31.420456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:31.420403 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:33.086551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:33.085667 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:33.264169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:33.264140 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:34.552293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:34.552259 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:34.724693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:34.724661 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:35.312133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:35.312091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:35.743583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:35.743486 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:36.556769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:36.556731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:36.713951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:36.713918 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:37.507729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.507691 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x"] Nov 03 17:24:37.509383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509349 4539 cpu_manager.go:414] "containerMap: removing container" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" Nov 03 17:24:37.509383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509381 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509406 4539 cpu_manager.go:414] "containerMap: removing container" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509414 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509446 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9da67c33-2e77-4f6b-8897-ab346420a222" containerName="main" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509453 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da67c33-2e77-4f6b-8897-ab346420a222" containerName="main" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509470 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f2309506-25e4-4d6b-a718-11a3a5991386" containerName="main" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509476 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2309506-25e4-4d6b-a718-11a3a5991386" containerName="main" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509503 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1329b928-6285-4d41-8a3e-af0b55343c16" containerName="discovery" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509510 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1329b928-6285-4d41-8a3e-af0b55343c16" containerName="discovery" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509525 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0dd9deaa-49be-411d-9b7d-7d9722f9918f" containerName="main" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509531 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd9deaa-49be-411d-9b7d-7d9722f9918f" containerName="main" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509545 4539 cpu_manager.go:414] "containerMap: removing container" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" Nov 03 17:24:37.509598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509553 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509713 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56e708e-f97c-44b7-97bd-c31970eab99c" containerName="istio-proxy" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509724 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="11212458-38d0-48bd-a512-ecb6b9b24dc1" containerName="istio-proxy" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509738 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da67c33-2e77-4f6b-8897-ab346420a222" containerName="main" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509747 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2309506-25e4-4d6b-a718-11a3a5991386" containerName="main" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509759 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1329b928-6285-4d41-8a3e-af0b55343c16" containerName="discovery" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509767 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd9deaa-49be-411d-9b7d-7d9722f9918f" containerName="main" Nov 03 17:24:37.510028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.509777 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="24266884-b424-4358-9359-594731ec8628" containerName="istio-proxy" Nov 03 17:24:37.515717 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.515687 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.538262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.538225 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x"] Nov 03 17:24:37.541410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.541381 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-527051-openshift-default-dockercfg-wcccz\"" Nov 03 17:24:37.694146 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694103 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694170 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694252 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694307 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhtcs\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694374 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694482 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694516 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694540 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.694676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.694604 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.796493 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.796459 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.796711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.796522 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.796711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.796566 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.796711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.796623 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhtcs\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.796884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.796712 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.797076 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.797091 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.797666 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.796751 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.797747 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.797771 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.797834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.797788 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.798108 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.798085 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.798199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.798177 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.799548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.799515 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.800084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.800060 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.819896 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.819867 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.820074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.820003 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhtcs\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:37.831865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:37.831838 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:38.157758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.157728 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x"] Nov 03 17:24:38.158067 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:24:38.158041 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5605eca6_fcbd_4702_b85e_54ec28aa86a2.slice/crio-3156872c44858611627d87d0d483aff6468152bec855670436fff000cc191c9e WatchSource:0}: Error finding container 3156872c44858611627d87d0d483aff6468152bec855670436fff000cc191c9e: Status 404 returned error can't find the container with id 3156872c44858611627d87d0d483aff6468152bec855670436fff000cc191c9e Nov 03 17:24:38.161278 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.161247 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:24:38.161383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.161315 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:24:38.161383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.161343 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:24:38.488718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.488626 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:38.746734 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.746652 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:38.953161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.953125 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" event={"ID":"5605eca6-fcbd-4702-b85e-54ec28aa86a2","Type":"ContainerStarted","Data":"4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567"} Nov 03 17:24:38.953386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:38.953207 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" event={"ID":"5605eca6-fcbd-4702-b85e-54ec28aa86a2","Type":"ContainerStarted","Data":"3156872c44858611627d87d0d483aff6468152bec855670436fff000cc191c9e"} Nov 03 17:24:39.042449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:39.025914 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podStartSLOduration=2.025893673 podStartE2EDuration="2.025893673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:24:39.012161161 +0000 UTC m=+4222.705432185" watchObservedRunningTime="2026-11-03 17:24:39.025893673 +0000 UTC m=+4222.719164695" Nov 03 17:24:39.833450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:39.832629 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:39.834954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:39.834527 4539 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 03 17:24:39.834954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:39.834586 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 03 17:24:40.209106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:40.209010 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:40.832926 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:40.832888 4539 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 03 17:24:40.833242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:40.833210 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 03 17:24:41.714913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:41.714878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:41.834938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:41.834899 4539 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 03 17:24:41.835290 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:41.835252 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 03 17:24:42.832908 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:42.832871 4539 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 03 17:24:42.833551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:42.833520 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 03 17:24:43.395279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:43.395246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:43.553275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:43.553243 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:43.832633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:43.832595 4539 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 03 17:24:43.832844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:43.832675 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 03 17:24:44.034388 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:44.034353 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:44.182711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:44.182621 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:45.023147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:45.022418 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:45.024095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:45.024069 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:45.025349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:45.025327 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:24:45.571273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:45.571235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:47.382416 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:47.382378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:48.550001 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:48.549962 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675"] Nov 03 17:24:48.550599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:48.550161 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" containerID="cri-o://ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241" gracePeriod=30 Nov 03 17:24:49.024515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:49.024452 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:50.530038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:50.530002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:50.714041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:50.714008 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:52.451386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:52.451352 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:53.846343 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.846308 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:24:53.932103 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.932067 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:53.961697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.961668 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.961904 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.961724 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.961904 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.961875 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.962042 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.961954 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.962042 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.961974 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.962042 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.961997 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.962042 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.962026 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.962281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.962094 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.962281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.962130 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo\") pod \"56c7d2f5-ee54-4e58-9f7d-48987054010f\" (UID: \"56c7d2f5-ee54-4e58-9f7d-48987054010f\") " Nov 03 17:24:53.963194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.962689 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:53.963194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.962925 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:53.963194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.962947 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:53.963194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.963166 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:53.963617 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.963585 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:53.963710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.963661 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:53.966288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.966261 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:53.966914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.966894 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "kube-api-access-nvt8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:53.967159 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.967137 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:53.967252 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:53.967177 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token" pod "56c7d2f5-ee54-4e58-9f7d-48987054010f" (UID: "56c7d2f5-ee54-4e58-9f7d-48987054010f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:54.064962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.064926 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c7d2f5-ee54-4e58-9f7d-48987054010f-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.064962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.064960 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.065181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.064970 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.065181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.064981 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.065181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.064990 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvt8t\" (UniqueName: \"kubernetes.io/projected/56c7d2f5-ee54-4e58-9f7d-48987054010f-kube-api-access-nvt8t\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.065181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.064999 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.065181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.065007 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.065181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.065015 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c7d2f5-ee54-4e58-9f7d-48987054010f-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:24:54.088795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.088700 4539 generic.go:356] "Generic (PLEG): container finished" podID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerID="ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241" exitCode=0 Nov 03 17:24:54.088795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.088755 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" Nov 03 17:24:54.089013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.088802 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" event={"ID":"56c7d2f5-ee54-4e58-9f7d-48987054010f","Type":"ContainerDied","Data":"ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241"} Nov 03 17:24:54.089013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.088873 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675" event={"ID":"56c7d2f5-ee54-4e58-9f7d-48987054010f","Type":"ContainerDied","Data":"4b3d4754044e916f0819e4ba082decf5b1a49114ef27ff6c2887e68c3b98af39"} Nov 03 17:24:54.089013 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.088897 4539 scope.go:122] "RemoveContainer" containerID="ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241" Nov 03 17:24:54.109661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.109628 4539 scope.go:122] "RemoveContainer" containerID="ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241" Nov 03 17:24:54.110066 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:24:54.110037 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241\": container with ID starting with ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241 not found: ID does not exist" containerID="ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241" Nov 03 17:24:54.110152 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.110073 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241"} err="failed to get container status \"ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241\": rpc error: code = NotFound desc = could not find container \"ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241\": container with ID starting with ee886d8556cff1b5f4ba9ef329a3f686a76b34e48f4c52acc2b1d110f32e5241 not found: ID does not exist" Nov 03 17:24:54.146914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.146874 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675"] Nov 03 17:24:54.151402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.151374 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rl675"] Nov 03 17:24:54.471288 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:54.471207 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" path="/var/lib/kubelet/pods/56c7d2f5-ee54-4e58-9f7d-48987054010f/volumes" Nov 03 17:24:55.610920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:55.610853 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:56.160414 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:24:56.160371 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:24:56.662642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:56.662611 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629"] Nov 03 17:24:56.745846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:56.745814 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:57.413131 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:57.413088 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:24:59.091379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:24:59.091342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:00.663211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:00.663175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:02.262616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:02.262584 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:03.594073 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:03.594027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:04.455077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:04.455029 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:06.059676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:06.059638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:07.685605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:07.685568 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:09.243672 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:09.243634 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:09.429861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:09.429825 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:09.598437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:09.598388 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:10.164187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:10.164155 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:10.554051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:10.554007 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:11.130817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:11.130765 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:11.351320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:11.351281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:11.715810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:11.715751 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" podUID="a3667ce2-6425-4264-b6bc-592f6272b109" containerName="main" containerID="cri-o://feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff" gracePeriod=45 Nov 03 17:25:12.012767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.012738 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:25:12.131787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.131756 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir\") pod \"a3667ce2-6425-4264-b6bc-592f6272b109\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " Nov 03 17:25:12.131967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.131895 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache\") pod \"a3667ce2-6425-4264-b6bc-592f6272b109\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " Nov 03 17:25:12.131967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.131949 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs\") pod \"a3667ce2-6425-4264-b6bc-592f6272b109\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " Nov 03 17:25:12.132114 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.131999 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home\") pod \"a3667ce2-6425-4264-b6bc-592f6272b109\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " Nov 03 17:25:12.132114 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132086 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm\") pod \"a3667ce2-6425-4264-b6bc-592f6272b109\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " Nov 03 17:25:12.132251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132108 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir" pod "a3667ce2-6425-4264-b6bc-592f6272b109" (UID: "a3667ce2-6425-4264-b6bc-592f6272b109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:12.132251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132136 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv\") pod \"a3667ce2-6425-4264-b6bc-592f6272b109\" (UID: \"a3667ce2-6425-4264-b6bc-592f6272b109\") " Nov 03 17:25:12.132251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132136 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache" pod "a3667ce2-6425-4264-b6bc-592f6272b109" (UID: "a3667ce2-6425-4264-b6bc-592f6272b109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:12.132439 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132336 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home" pod "a3667ce2-6425-4264-b6bc-592f6272b109" (UID: "a3667ce2-6425-4264-b6bc-592f6272b109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:12.132698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132672 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:12.132698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132700 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:12.132698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.132713 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:12.134529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.134503 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs" pod "a3667ce2-6425-4264-b6bc-592f6272b109" (UID: "a3667ce2-6425-4264-b6bc-592f6272b109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:12.134649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.134586 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv" pod "a3667ce2-6425-4264-b6bc-592f6272b109" (UID: "a3667ce2-6425-4264-b6bc-592f6272b109"). InnerVolumeSpecName "kube-api-access-gbplv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:12.134649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.134613 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm" pod "a3667ce2-6425-4264-b6bc-592f6272b109" (UID: "a3667ce2-6425-4264-b6bc-592f6272b109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:12.234711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.234627 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3667ce2-6425-4264-b6bc-592f6272b109-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:12.234711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.234654 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3667ce2-6425-4264-b6bc-592f6272b109-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:12.234711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.234663 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbplv\" (UniqueName: \"kubernetes.io/projected/a3667ce2-6425-4264-b6bc-592f6272b109-kube-api-access-gbplv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:12.251956 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.251922 4539 generic.go:356] "Generic (PLEG): container finished" podID="a3667ce2-6425-4264-b6bc-592f6272b109" containerID="feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff" exitCode=0 Nov 03 17:25:12.252165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.251977 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" Nov 03 17:25:12.252165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.252011 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" event={"ID":"a3667ce2-6425-4264-b6bc-592f6272b109","Type":"ContainerDied","Data":"feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff"} Nov 03 17:25:12.252165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.252066 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629" event={"ID":"a3667ce2-6425-4264-b6bc-592f6272b109","Type":"ContainerDied","Data":"cbdd26da00e0fef82224f2396077e3653f3c428e6d4c89fa3a3902351d260d5a"} Nov 03 17:25:12.252165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.252089 4539 scope.go:122] "RemoveContainer" containerID="feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff" Nov 03 17:25:12.263971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.263949 4539 scope.go:122] "RemoveContainer" containerID="feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff" Nov 03 17:25:12.264288 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:25:12.264268 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff\": container with ID starting with feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff not found: ID does not exist" containerID="feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff" Nov 03 17:25:12.264363 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.264311 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff"} err="failed to get container status \"feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff\": rpc error: code = NotFound desc = could not find container \"feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff\": container with ID starting with feb41b9f8abcd99b4162797b2437fd5891840b68fa283fe131b873acb7e4c0ff not found: ID does not exist" Nov 03 17:25:12.301659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.301624 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629"] Nov 03 17:25:12.306046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.306020 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-44736d/test-model-44736d-kserve-67cbfc6fc5-5p629"] Nov 03 17:25:12.471724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.471685 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3667ce2-6425-4264-b6bc-592f6272b109" path="/var/lib/kubelet/pods/a3667ce2-6425-4264-b6bc-592f6272b109/volumes" Nov 03 17:25:12.921502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:12.921264 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:14.010411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:14.010291 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:14.711728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:14.711688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:16.376769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:16.376734 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:16.538676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:16.538633 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:16.766620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:16.766542 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:16.935881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:16.935844 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:18.276987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:18.276952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:18.450722 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:18.450689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:19.676829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:19.676777 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq"] Nov 03 17:25:19.758030 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:19.757990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:20.032967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:20.032926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:20.188804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:20.188774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:21.593578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:21.593544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:23.279695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:23.278613 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:24.531764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:24.531728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:24.874917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:24.874827 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:26.159520 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:25:26.159479 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:25:26.710229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:26.710197 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:26.851374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:26.851340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:27.013164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:27.013080 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:27.743611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:27.743582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:28.439571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:28.439535 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:29.979521 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:29.979481 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:30.144872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:30.144839 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:30.377454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:30.377388 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:31.705340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:31.705307 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:33.084638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:33.084602 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:33.649501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:33.649464 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:33.904862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:33.904588 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:34.712394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:34.712325 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" podUID="428f9575-cc5c-4cb1-87b8-cc851778cdea" containerName="main" containerID="cri-o://b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3" gracePeriod=45 Nov 03 17:25:34.789134 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:34.789101 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:34.948973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:34.948834 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:35.023220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.023189 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:25:35.134357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.134318 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache\") pod \"428f9575-cc5c-4cb1-87b8-cc851778cdea\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " Nov 03 17:25:35.134619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.134491 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw\" (UniqueName: \"kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw\") pod \"428f9575-cc5c-4cb1-87b8-cc851778cdea\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " Nov 03 17:25:35.134699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.134623 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm\") pod \"428f9575-cc5c-4cb1-87b8-cc851778cdea\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " Nov 03 17:25:35.134699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.134650 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache" pod "428f9575-cc5c-4cb1-87b8-cc851778cdea" (UID: "428f9575-cc5c-4cb1-87b8-cc851778cdea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:35.134699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.134680 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs\" (UniqueName: \"kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs\") pod \"428f9575-cc5c-4cb1-87b8-cc851778cdea\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " Nov 03 17:25:35.134873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.134733 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir\") pod \"428f9575-cc5c-4cb1-87b8-cc851778cdea\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " Nov 03 17:25:35.135115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.135025 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir" pod "428f9575-cc5c-4cb1-87b8-cc851778cdea" (UID: "428f9575-cc5c-4cb1-87b8-cc851778cdea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:35.135251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.135177 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home\") pod \"428f9575-cc5c-4cb1-87b8-cc851778cdea\" (UID: \"428f9575-cc5c-4cb1-87b8-cc851778cdea\") " Nov 03 17:25:35.135592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.135551 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home" pod "428f9575-cc5c-4cb1-87b8-cc851778cdea" (UID: "428f9575-cc5c-4cb1-87b8-cc851778cdea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:35.135940 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.135920 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:35.136050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.135944 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:35.136050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.135957 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:35.137566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.137543 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm" pod "428f9575-cc5c-4cb1-87b8-cc851778cdea" (UID: "428f9575-cc5c-4cb1-87b8-cc851778cdea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:35.137715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.137660 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs" pod "428f9575-cc5c-4cb1-87b8-cc851778cdea" (UID: "428f9575-cc5c-4cb1-87b8-cc851778cdea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:35.137715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.137709 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw" pod "428f9575-cc5c-4cb1-87b8-cc851778cdea" (UID: "428f9575-cc5c-4cb1-87b8-cc851778cdea"). InnerVolumeSpecName "kube-api-access-892rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:35.237322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.237223 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892rw\" (UniqueName: \"kubernetes.io/projected/428f9575-cc5c-4cb1-87b8-cc851778cdea-kube-api-access-892rw\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:35.237322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.237258 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428f9575-cc5c-4cb1-87b8-cc851778cdea-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:35.237322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.237267 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428f9575-cc5c-4cb1-87b8-cc851778cdea-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:35.533217 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.533170 4539 generic.go:356] "Generic (PLEG): container finished" podID="428f9575-cc5c-4cb1-87b8-cc851778cdea" containerID="b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3" exitCode=0 Nov 03 17:25:35.533466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.533216 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" Nov 03 17:25:35.533466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.533229 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" event={"ID":"428f9575-cc5c-4cb1-87b8-cc851778cdea","Type":"ContainerDied","Data":"b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3"} Nov 03 17:25:35.533466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.533277 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq" event={"ID":"428f9575-cc5c-4cb1-87b8-cc851778cdea","Type":"ContainerDied","Data":"6ec856d81637cbde6a65d2f470b29997a2f4b036225c5f54bf5042070681ba86"} Nov 03 17:25:35.533466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.533299 4539 scope.go:122] "RemoveContainer" containerID="b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3" Nov 03 17:25:35.547567 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.547519 4539 scope.go:122] "RemoveContainer" containerID="b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3" Nov 03 17:25:35.547884 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:25:35.547854 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3\": container with ID starting with b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3 not found: ID does not exist" containerID="b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3" Nov 03 17:25:35.547960 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.547899 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3"} err="failed to get container status \"b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3\": rpc error: code = NotFound desc = could not find container \"b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3\": container with ID starting with b89beeca2878ed86839cef53ca5833ae30cb0210ed9f9efdccc67626f695f2c3 not found: ID does not exist" Nov 03 17:25:35.566655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.566626 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:35.582355 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.582328 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq"] Nov 03 17:25:35.593947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.593091 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-236539/test-model-236539-kserve-96f68f49f-r9slq"] Nov 03 17:25:35.718089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.718054 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:35.954342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:35.954255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:36.393974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:36.393942 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:36.471834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:36.471801 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428f9575-cc5c-4cb1-87b8-cc851778cdea" path="/var/lib/kubelet/pods/428f9575-cc5c-4cb1-87b8-cc851778cdea/volumes" Nov 03 17:25:36.642012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:36.641981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:36.853349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:36.853318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:37.028765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:37.028720 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:37.443559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:37.443525 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:37.730737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:37.730652 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:38.145255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:38.145217 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:38.290568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:38.290524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:38.570142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:38.570107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:38.732618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:38.732574 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:39.330138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:39.330101 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:40.690296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:40.690256 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm"] Nov 03 17:25:40.690854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:40.690528 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" podUID="4c791259-d616-4cd1-8def-c481def1035f" containerName="maas-api" containerID="cri-o://f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2" gracePeriod=30 Nov 03 17:25:40.807010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:40.806976 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:41.428819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.428791 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:25:41.574394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.574361 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd\" (UniqueName: \"kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd\") pod \"4c791259-d616-4cd1-8def-c481def1035f\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " Nov 03 17:25:41.574636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.574548 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca\") pod \"4c791259-d616-4cd1-8def-c481def1035f\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " Nov 03 17:25:41.574636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.574620 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\") pod \"4c791259-d616-4cd1-8def-c481def1035f\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " Nov 03 17:25:41.574769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.574673 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs\") pod \"4c791259-d616-4cd1-8def-c481def1035f\" (UID: \"4c791259-d616-4cd1-8def-c481def1035f\") " Nov 03 17:25:41.574967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.574939 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca" pod "4c791259-d616-4cd1-8def-c481def1035f" (UID: "4c791259-d616-4cd1-8def-c481def1035f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:41.575341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.575321 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c791259-d616-4cd1-8def-c481def1035f-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.576845 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.576815 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs" pod "4c791259-d616-4cd1-8def-c481def1035f" (UID: "4c791259-d616-4cd1-8def-c481def1035f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:41.576934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.576883 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls" pod "4c791259-d616-4cd1-8def-c481def1035f" (UID: "4c791259-d616-4cd1-8def-c481def1035f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:41.576934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.576901 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd" pod "4c791259-d616-4cd1-8def-c481def1035f" (UID: "4c791259-d616-4cd1-8def-c481def1035f"). InnerVolumeSpecName "kube-api-access-b5czd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:41.585465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.585402 4539 generic.go:356] "Generic (PLEG): container finished" podID="4c791259-d616-4cd1-8def-c481def1035f" containerID="f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2" exitCode=0 Nov 03 17:25:41.585613 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.585477 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" Nov 03 17:25:41.585613 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.585572 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" event={"ID":"4c791259-d616-4cd1-8def-c481def1035f","Type":"ContainerDied","Data":"f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2"} Nov 03 17:25:41.585736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.585626 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm" event={"ID":"4c791259-d616-4cd1-8def-c481def1035f","Type":"ContainerDied","Data":"86a9411dfd134b5c4f46e037cb730ccadaf597509cd912c1cde49f3d61e664c4"} Nov 03 17:25:41.585736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.585652 4539 scope.go:122] "RemoveContainer" containerID="f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2" Nov 03 17:25:41.602197 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.602169 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:41.612059 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.609794 4539 scope.go:122] "RemoveContainer" containerID="f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2" Nov 03 17:25:41.612059 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:25:41.610189 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2\": container with ID starting with f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2 not found: ID does not exist" containerID="f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2" Nov 03 17:25:41.612059 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.610221 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2"} err="failed to get container status \"f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2\": rpc error: code = NotFound desc = could not find container \"f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2\": container with ID starting with f6c4a30b882793de8291b5ecb23086166560ab8df209f0ad01648a4d6d9195c2 not found: ID does not exist" Nov 03 17:25:41.684924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.684775 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.684924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.684816 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c791259-d616-4cd1-8def-c481def1035f-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.684924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.684830 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5czd\" (UniqueName: \"kubernetes.io/projected/4c791259-d616-4cd1-8def-c481def1035f-kube-api-access-b5czd\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.688450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.685476 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm"] Nov 03 17:25:41.750455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:41.742713 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb0fa3-54669646f-w9zpm"] Nov 03 17:25:42.469980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:42.469946 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c791259-d616-4cd1-8def-c481def1035f" path="/var/lib/kubelet/pods/4c791259-d616-4cd1-8def-c481def1035f/volumes" Nov 03 17:25:43.461460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:43.461385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:44.528997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:44.528957 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:44.705085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:44.705055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:45.372075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:45.372018 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:45.528383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:45.528355 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:45.719271 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:45.719188 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:46.546296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:46.546261 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:47.554285 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:47.554249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:49.188642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:49.188604 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:50.831940 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:50.831904 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:52.485139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:52.485107 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:52.632101 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:52.632072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:53.193909 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:53.193869 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:54.265161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:54.265128 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:55.873112 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:55.873071 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:56.158792 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:25:56.158707 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:25:57.495136 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:57.495100 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:25:59.179268 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:25:59.179231 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:00.074138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:00.074102 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:00.888252 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:00.888206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:02.487534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:02.487491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:03.785321 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:03.785290 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:04.071034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:04.070933 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:05.623279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:05.623245 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:07.319898 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:07.319851 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:08.676156 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:08.676124 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:09.168255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:09.168212 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:09.886358 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:09.886309 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:10.624148 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:10.624093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:10.768831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:10.768774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:11.558833 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:11.558801 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:11.714246 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:11.714213 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:11.870138 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:11.870042 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x"] Nov 03 17:26:11.870676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:11.870639 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" containerID="cri-o://4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567" gracePeriod=30 Nov 03 17:26:11.939087 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:11.939046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:12.925934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:12.925841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:13.071237 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:13.071206 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:13.289292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:13.289244 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:14.923836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:14.923799 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:15.026213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:15.026176 4539 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:26:15.026441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:15.026240 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:26:15.550666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:15.550632 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:15.743337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:15.743299 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:16.626976 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:16.626940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:16.780004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:16.779961 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:16.811318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:16.811278 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84"] Nov 03 17:26:16.939171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:16.939100 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:17.232044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.232014 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:26:17.340241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340189 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340490 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340321 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340490 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340391 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340562 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340618 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340661 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340695 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340739 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.340917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.340856 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data\") pod \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\" (UID: \"5605eca6-fcbd-4702-b85e-54ec28aa86a2\") " Nov 03 17:26:17.341293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.341268 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.341647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.341627 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.343294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.343244 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:17.343486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.343450 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.343803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.343765 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.344814 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.344788 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.344907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.344819 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.344907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.344833 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.344907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.344845 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.344907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.344858 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.345133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.344990 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "kube-api-access-rhtcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:17.350917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.349221 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:17.351169 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.351140 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:26:17.351299 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.351274 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy" pod "5605eca6-fcbd-4702-b85e-54ec28aa86a2" (UID: "5605eca6-fcbd-4702-b85e-54ec28aa86a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.450762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.450667 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.450762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.450712 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.450762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.450728 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhtcs\" (UniqueName: \"kubernetes.io/projected/5605eca6-fcbd-4702-b85e-54ec28aa86a2-kube-api-access-rhtcs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.450762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.450744 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5605eca6-fcbd-4702-b85e-54ec28aa86a2-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.883093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.883054 4539 generic.go:356] "Generic (PLEG): container finished" podID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerID="4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567" exitCode=0 Nov 03 17:26:17.883578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.883113 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" Nov 03 17:26:17.883578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.883134 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" event={"ID":"5605eca6-fcbd-4702-b85e-54ec28aa86a2","Type":"ContainerDied","Data":"4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567"} Nov 03 17:26:17.883578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.883200 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x" event={"ID":"5605eca6-fcbd-4702-b85e-54ec28aa86a2","Type":"ContainerDied","Data":"3156872c44858611627d87d0d483aff6468152bec855670436fff000cc191c9e"} Nov 03 17:26:17.883578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.883229 4539 scope.go:122] "RemoveContainer" containerID="4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567" Nov 03 17:26:17.895859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.895841 4539 scope.go:122] "RemoveContainer" containerID="4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567" Nov 03 17:26:17.896172 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:26:17.896144 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567\": container with ID starting with 4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567 not found: ID does not exist" containerID="4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567" Nov 03 17:26:17.896272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.896178 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567"} err="failed to get container status \"4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567\": rpc error: code = NotFound desc = could not find container \"4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567\": container with ID starting with 4eaf5ca3952a08fdc68ab28fed5a89ea25d0f0f9e83debcd5a8fa6a032d45567 not found: ID does not exist" Nov 03 17:26:17.933573 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.933533 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x"] Nov 03 17:26:17.943804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:17.943773 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-qlh8x"] Nov 03 17:26:18.129190 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:18.129138 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:18.281508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:18.281477 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:18.469935 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:18.469894 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" path="/var/lib/kubelet/pods/5605eca6-fcbd-4702-b85e-54ec28aa86a2/volumes" Nov 03 17:26:18.589137 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:18.589032 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:18.837345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:18.837307 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:19.188687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:19.188647 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:19.956129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:19.956090 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:20.460151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:20.460115 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:22.085730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:22.085670 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:23.698867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:23.698810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:25.353634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:25.353601 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:26.294704 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:26:26.294661 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:26:26.886574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:26.886533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:27.056061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:27.056025 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:27.202920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:27.202837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:28.563343 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:28.563309 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:29.299215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:29.299186 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:30.933219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:30.933180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:31.845907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:31.845833 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" podUID="144603bc-fbc4-4006-91ff-7f08ab5d1769" containerName="main" containerID="cri-o://dffdb10be8f1cfdcfdbf226533a76d8a5e84a33f2d2e2a69778598b76c8d9a5b" gracePeriod=45 Nov 03 17:26:31.866720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:31.866675 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" podUID="144603bc-fbc4-4006-91ff-7f08ab5d1769" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": dial tcp 10.133.0.196:8000: connect: connection refused" Nov 03 17:26:31.868884 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:26:31.868854 4539 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod144603bc_fbc4_4006_91ff_7f08ab5d1769.slice/crio-dffdb10be8f1cfdcfdbf226533a76d8a5e84a33f2d2e2a69778598b76c8d9a5b.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:26:31.996520 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:31.996489 4539 generic.go:356] "Generic (PLEG): container finished" podID="144603bc-fbc4-4006-91ff-7f08ab5d1769" containerID="dffdb10be8f1cfdcfdbf226533a76d8a5e84a33f2d2e2a69778598b76c8d9a5b" exitCode=0 Nov 03 17:26:31.996928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:31.996559 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" event={"ID":"144603bc-fbc4-4006-91ff-7f08ab5d1769","Type":"ContainerDied","Data":"dffdb10be8f1cfdcfdbf226533a76d8a5e84a33f2d2e2a69778598b76c8d9a5b"} Nov 03 17:26:32.178509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.178481 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:26:32.297396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.297360 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5\" (UniqueName: \"kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5\") pod \"144603bc-fbc4-4006-91ff-7f08ab5d1769\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " Nov 03 17:26:32.297627 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.297514 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs\" (UniqueName: \"kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs\") pod \"144603bc-fbc4-4006-91ff-7f08ab5d1769\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " Nov 03 17:26:32.297711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.297694 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir\") pod \"144603bc-fbc4-4006-91ff-7f08ab5d1769\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " Nov 03 17:26:32.297777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.297751 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm\") pod \"144603bc-fbc4-4006-91ff-7f08ab5d1769\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " Nov 03 17:26:32.297840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.297792 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache\") pod \"144603bc-fbc4-4006-91ff-7f08ab5d1769\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " Nov 03 17:26:32.297903 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.297873 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home\") pod \"144603bc-fbc4-4006-91ff-7f08ab5d1769\" (UID: \"144603bc-fbc4-4006-91ff-7f08ab5d1769\") " Nov 03 17:26:32.298039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.298007 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir" pod "144603bc-fbc4-4006-91ff-7f08ab5d1769" (UID: "144603bc-fbc4-4006-91ff-7f08ab5d1769"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:32.298464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.298404 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache" pod "144603bc-fbc4-4006-91ff-7f08ab5d1769" (UID: "144603bc-fbc4-4006-91ff-7f08ab5d1769"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:32.298464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.298418 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home" pod "144603bc-fbc4-4006-91ff-7f08ab5d1769" (UID: "144603bc-fbc4-4006-91ff-7f08ab5d1769"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:32.298784 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.298760 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.298869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.298793 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.298869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.298807 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.300205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.300175 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs" pod "144603bc-fbc4-4006-91ff-7f08ab5d1769" (UID: "144603bc-fbc4-4006-91ff-7f08ab5d1769"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:32.300205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.300192 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5" pod "144603bc-fbc4-4006-91ff-7f08ab5d1769" (UID: "144603bc-fbc4-4006-91ff-7f08ab5d1769"). InnerVolumeSpecName "kube-api-access-b76w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:32.300347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.300294 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm" pod "144603bc-fbc4-4006-91ff-7f08ab5d1769" (UID: "144603bc-fbc4-4006-91ff-7f08ab5d1769"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:32.400119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.400024 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144603bc-fbc4-4006-91ff-7f08ab5d1769-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.400119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.400058 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b76w5\" (UniqueName: \"kubernetes.io/projected/144603bc-fbc4-4006-91ff-7f08ab5d1769-kube-api-access-b76w5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.400119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.400074 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144603bc-fbc4-4006-91ff-7f08ab5d1769-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.600581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:32.600517 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:33.005460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:33.005382 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" event={"ID":"144603bc-fbc4-4006-91ff-7f08ab5d1769","Type":"ContainerDied","Data":"d535a839ddf4c2de0d8c6bce9d64bf503975e9b8d3a252c04386e894b92baad3"} Nov 03 17:26:33.005961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:33.005480 4539 scope.go:122] "RemoveContainer" containerID="dffdb10be8f1cfdcfdbf226533a76d8a5e84a33f2d2e2a69778598b76c8d9a5b" Nov 03 17:26:33.005961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:33.005401 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84" Nov 03 17:26:33.052793 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:33.052755 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84"] Nov 03 17:26:33.063357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:33.063324 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-472bee/test-model-472bee-kserve-578d88976f-mkh84"] Nov 03 17:26:34.278463 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:34.277544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:34.470212 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:34.470176 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144603bc-fbc4-4006-91ff-7f08ab5d1769" path="/var/lib/kubelet/pods/144603bc-fbc4-4006-91ff-7f08ab5d1769/volumes" Nov 03 17:26:35.068074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:35.068034 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:35.827507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:35.827456 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:37.235620 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:37.235583 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:37.420483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:37.420381 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:37.644093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:37.644050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:38.770781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:38.770743 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:39.460808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:39.460716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:40.951942 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:40.951716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:42.495810 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:42.495773 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:43.534382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:43.534345 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:43.734152 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:43.734118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:44.383080 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:44.383024 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:44.738499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:44.738384 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9"] Nov 03 17:26:44.739311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:44.739270 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" containerID="cri-o://cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa" gracePeriod=30 Nov 03 17:26:46.120243 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:46.120204 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:46.348201 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:46.348164 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:47.791303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:47.791267 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc"] Nov 03 17:26:47.791752 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:47.791567 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" podUID="41c73247-c9ce-408e-8569-d1eb2eaaa77d" containerName="maas-api" containerID="cri-o://a89d2a58cdeda8ae14c994e2f3a95dd9fd229701a75769d19ce7c5c2a32f6155" gracePeriod=30 Nov 03 17:26:47.937236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:47.937194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:48.172677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.172476 4539 generic.go:356] "Generic (PLEG): container finished" podID="41c73247-c9ce-408e-8569-d1eb2eaaa77d" containerID="a89d2a58cdeda8ae14c994e2f3a95dd9fd229701a75769d19ce7c5c2a32f6155" exitCode=0 Nov 03 17:26:48.172677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.172604 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" event={"ID":"41c73247-c9ce-408e-8569-d1eb2eaaa77d","Type":"ContainerDied","Data":"a89d2a58cdeda8ae14c994e2f3a95dd9fd229701a75769d19ce7c5c2a32f6155"} Nov 03 17:26:48.496144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.496118 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:26:48.498252 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.498227 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls\") pod \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " Nov 03 17:26:48.498381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.498321 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj\" (UniqueName: \"kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj\") pod \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " Nov 03 17:26:48.498381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.498362 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs\") pod \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " Nov 03 17:26:48.498495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.498412 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca\") pod \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\" (UID: \"41c73247-c9ce-408e-8569-d1eb2eaaa77d\") " Nov 03 17:26:48.498881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.498845 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca" pod "41c73247-c9ce-408e-8569-d1eb2eaaa77d" (UID: "41c73247-c9ce-408e-8569-d1eb2eaaa77d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:48.499182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.499093 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c73247-c9ce-408e-8569-d1eb2eaaa77d-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:48.501032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.500991 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls" pod "41c73247-c9ce-408e-8569-d1eb2eaaa77d" (UID: "41c73247-c9ce-408e-8569-d1eb2eaaa77d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:48.501377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.501348 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj" pod "41c73247-c9ce-408e-8569-d1eb2eaaa77d" (UID: "41c73247-c9ce-408e-8569-d1eb2eaaa77d"). InnerVolumeSpecName "kube-api-access-zzwqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:48.501505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.501382 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs" pod "41c73247-c9ce-408e-8569-d1eb2eaaa77d" (UID: "41c73247-c9ce-408e-8569-d1eb2eaaa77d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:48.600167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.600137 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:48.600167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.600166 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwqj\" (UniqueName: \"kubernetes.io/projected/41c73247-c9ce-408e-8569-d1eb2eaaa77d-kube-api-access-zzwqj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:48.600167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.600177 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c73247-c9ce-408e-8569-d1eb2eaaa77d-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:48.754977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:48.754883 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:49.182403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:49.182370 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" Nov 03 17:26:49.182990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:49.182409 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc" event={"ID":"41c73247-c9ce-408e-8569-d1eb2eaaa77d","Type":"ContainerDied","Data":"d8fb92a065a2ffb06bea04c12500ce9c8ad0e5122bd237bf11bb5f7444994a62"} Nov 03 17:26:49.182990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:49.182497 4539 scope.go:122] "RemoveContainer" containerID="a89d2a58cdeda8ae14c994e2f3a95dd9fd229701a75769d19ce7c5c2a32f6155" Nov 03 17:26:49.242594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:49.242551 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc"] Nov 03 17:26:49.246485 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:49.246456 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-472bee-644b5494b5-tc5lc"] Nov 03 17:26:50.042692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.042668 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:26:50.122848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.122803 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.122925 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.122996 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123054 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123108 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123138 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123170 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.123437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123398 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123626 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert\") pod \"ee28d928-7be1-4218-b50d-f1fd00a37eca\" (UID: \"ee28d928-7be1-4218-b50d-f1fd00a37eca\") " Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123895 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.123961 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.124282 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.124305 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.124330 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:50.124638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.124623 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.125170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.124680 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.126344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.126300 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:26:50.126841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.126809 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "kube-api-access-clzgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:50.127799 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.127764 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.128492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.128462 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token" pod "ee28d928-7be1-4218-b50d-f1fd00a37eca" (UID: "ee28d928-7be1-4218-b50d-f1fd00a37eca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:50.193344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.193311 4539 generic.go:356] "Generic (PLEG): container finished" podID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerID="cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa" exitCode=0 Nov 03 17:26:50.193805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.193375 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" Nov 03 17:26:50.193805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.193406 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" event={"ID":"ee28d928-7be1-4218-b50d-f1fd00a37eca","Type":"ContainerDied","Data":"cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa"} Nov 03 17:26:50.193805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.193517 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9" event={"ID":"ee28d928-7be1-4218-b50d-f1fd00a37eca","Type":"ContainerDied","Data":"4b530192156239b2fcdb15417648c7cfeb848bfb2426f469b39ad874cb40f6bc"} Nov 03 17:26:50.193805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.193549 4539 scope.go:122] "RemoveContainer" containerID="cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa" Nov 03 17:26:50.207395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.207375 4539 scope.go:122] "RemoveContainer" containerID="61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5" Nov 03 17:26:50.220215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.220196 4539 scope.go:122] "RemoveContainer" containerID="cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa" Nov 03 17:26:50.220544 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:26:50.220524 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa\": container with ID starting with cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa not found: ID does not exist" containerID="cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa" Nov 03 17:26:50.220605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.220555 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa"} err="failed to get container status \"cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa\": rpc error: code = NotFound desc = could not find container \"cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa\": container with ID starting with cbbf4fa2bfc1302afb25e0f2f610299f5a0984aa5f304e25862570e748d9feaa not found: ID does not exist" Nov 03 17:26:50.220605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.220576 4539 scope.go:122] "RemoveContainer" containerID="61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5" Nov 03 17:26:50.220806 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:26:50.220790 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5\": container with ID starting with 61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5 not found: ID does not exist" containerID="61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5" Nov 03 17:26:50.220853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.220811 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5"} err="failed to get container status \"61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5\": rpc error: code = NotFound desc = could not find container \"61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5\": container with ID starting with 61748706441f0291af4dbee5a3cc2034c971f62ee3e37ae3b66783db2f3bf6c5 not found: ID does not exist" Nov 03 17:26:50.225215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225195 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28d928-7be1-4218-b50d-f1fd00a37eca-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.225215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225216 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.225349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225224 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzgk\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-kube-api-access-clzgk\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.225349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225233 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.225349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225242 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.225349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225250 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.225349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.225259 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28d928-7be1-4218-b50d-f1fd00a37eca-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.244778 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.244754 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9"] Nov 03 17:26:50.249588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.249562 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-hckt9"] Nov 03 17:26:50.323104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.323050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:50.470223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.470191 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c73247-c9ce-408e-8569-d1eb2eaaa77d" path="/var/lib/kubelet/pods/41c73247-c9ce-408e-8569-d1eb2eaaa77d/volumes" Nov 03 17:26:50.470787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:50.470770 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" path="/var/lib/kubelet/pods/ee28d928-7be1-4218-b50d-f1fd00a37eca/volumes" Nov 03 17:26:51.565563 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:51.565531 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:51.755992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:51.755959 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:52.072606 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:52.072570 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:52.295003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:52.294967 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:52.646332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:52.646297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:53.528818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:53.528771 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:53.807987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:53.807891 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:55.394319 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:55.394270 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:56.158260 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:26:56.158217 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:26:56.541598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:56.541564 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:57.028666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:57.028637 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:58.646565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:58.646334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:59.544728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:59.544693 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:26:59.703252 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:26:59.703222 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:00.349892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:00.349854 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:00.623222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:00.623138 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:02.209413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:02.209348 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:03.800153 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:03.800116 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:05.396785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:05.396709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:06.942874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:06.942840 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv"] Nov 03 17:27:06.943383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:06.943105 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" podUID="305e93ab-f9a0-4374-80a0-8c533b7684d4" containerName="maas-api" containerID="cri-o://0239b6e6952af758fa043c400c631adefd9000532b1f8301b004042e2e9c21cb" gracePeriod=30 Nov 03 17:27:07.031083 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.031041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:07.425786 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.425756 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:07.439851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.439502 4539 generic.go:356] "Generic (PLEG): container finished" podID="305e93ab-f9a0-4374-80a0-8c533b7684d4" containerID="0239b6e6952af758fa043c400c631adefd9000532b1f8301b004042e2e9c21cb" exitCode=0 Nov 03 17:27:07.439851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.439784 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" event={"ID":"305e93ab-f9a0-4374-80a0-8c533b7684d4","Type":"ContainerDied","Data":"0239b6e6952af758fa043c400c631adefd9000532b1f8301b004042e2e9c21cb"} Nov 03 17:27:07.473632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.473599 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:27:07.597941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.597905 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca\") pod \"305e93ab-f9a0-4374-80a0-8c533b7684d4\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " Nov 03 17:27:07.598148 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.598114 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p\" (UniqueName: \"kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p\") pod \"305e93ab-f9a0-4374-80a0-8c533b7684d4\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " Nov 03 17:27:07.598216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.598187 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs\") pod \"305e93ab-f9a0-4374-80a0-8c533b7684d4\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " Nov 03 17:27:07.598274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.598226 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\") pod \"305e93ab-f9a0-4374-80a0-8c533b7684d4\" (UID: \"305e93ab-f9a0-4374-80a0-8c533b7684d4\") " Nov 03 17:27:07.598329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.598276 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca" pod "305e93ab-f9a0-4374-80a0-8c533b7684d4" (UID: "305e93ab-f9a0-4374-80a0-8c533b7684d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:07.598869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.598842 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e93ab-f9a0-4374-80a0-8c533b7684d4-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:07.600679 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.600632 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p" pod "305e93ab-f9a0-4374-80a0-8c533b7684d4" (UID: "305e93ab-f9a0-4374-80a0-8c533b7684d4"). InnerVolumeSpecName "kube-api-access-9859p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:07.600771 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.600673 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls" pod "305e93ab-f9a0-4374-80a0-8c533b7684d4" (UID: "305e93ab-f9a0-4374-80a0-8c533b7684d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:27:07.600771 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.600718 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs" pod "305e93ab-f9a0-4374-80a0-8c533b7684d4" (UID: "305e93ab-f9a0-4374-80a0-8c533b7684d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:27:07.699918 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.699827 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9859p\" (UniqueName: \"kubernetes.io/projected/305e93ab-f9a0-4374-80a0-8c533b7684d4-kube-api-access-9859p\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:07.699918 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.699856 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:07.699918 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.699867 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305e93ab-f9a0-4374-80a0-8c533b7684d4-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:07.836588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:07.836558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:08.469515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:08.469396 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" Nov 03 17:27:08.478939 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:08.478889 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv" event={"ID":"305e93ab-f9a0-4374-80a0-8c533b7684d4","Type":"ContainerDied","Data":"7e8c9b9b9637db33c17138ac981d194ae2b474f9ed318276cc1764c759ab1787"} Nov 03 17:27:08.479254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:08.479232 4539 scope.go:122] "RemoveContainer" containerID="0239b6e6952af758fa043c400c631adefd9000532b1f8301b004042e2e9c21cb" Nov 03 17:27:08.548748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:08.548718 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv"] Nov 03 17:27:08.558091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:08.558043 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-236539-5459c8f8d7-cbfzv"] Nov 03 17:27:09.713020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:09.712986 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:10.392525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:10.392491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:10.471738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:10.471707 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305e93ab-f9a0-4374-80a0-8c533b7684d4" path="/var/lib/kubelet/pods/305e93ab-f9a0-4374-80a0-8c533b7684d4/volumes" Nov 03 17:27:10.570552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:10.570520 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:11.126174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:11.126136 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:11.483090 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:11.482985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:11.652937 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:11.652905 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:12.430152 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:12.430112 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:12.616263 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:12.616218 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:13.643860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:13.643826 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:15.229451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:15.223139 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:16.757713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:16.757669 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:16.974164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:16.974133 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:17.951999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:17.951955 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:18.488439 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:18.488393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:19.973269 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:19.973230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:21.561027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:21.560990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:23.086234 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:23.086201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:24.343990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:24.343956 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:24.797963 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:24.797929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:26.161352 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:27:26.161313 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:27:26.379995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:26.379956 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:26.977456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:26.977412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:28.034099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:28.034063 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:28.737325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:28.737274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:29.718849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:29.718798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:31.421789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:31.421726 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:32.885279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:32.885233 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:33.557392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.557358 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:33.988858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.988767 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg"] Nov 03 17:27:33.989998 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.989961 4539 cpu_manager.go:414] "containerMap: removing container" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" Nov 03 17:27:33.989998 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.989986 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990009 4539 cpu_manager.go:414] "containerMap: removing container" podUID="428f9575-cc5c-4cb1-87b8-cc851778cdea" containerName="main" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990016 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428f9575-cc5c-4cb1-87b8-cc851778cdea" containerName="main" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990031 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990040 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990054 4539 cpu_manager.go:414] "containerMap: removing container" podUID="41c73247-c9ce-408e-8569-d1eb2eaaa77d" containerName="maas-api" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990061 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c73247-c9ce-408e-8569-d1eb2eaaa77d" containerName="maas-api" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990075 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990082 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990097 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a3667ce2-6425-4264-b6bc-592f6272b109" containerName="main" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990104 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3667ce2-6425-4264-b6bc-592f6272b109" containerName="main" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990151 4539 cpu_manager.go:414] "containerMap: removing container" podUID="4c791259-d616-4cd1-8def-c481def1035f" containerName="maas-api" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990159 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c791259-d616-4cd1-8def-c481def1035f" containerName="maas-api" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990171 4539 cpu_manager.go:414] "containerMap: removing container" podUID="144603bc-fbc4-4006-91ff-7f08ab5d1769" containerName="main" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990178 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144603bc-fbc4-4006-91ff-7f08ab5d1769" containerName="main" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990190 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" Nov 03 17:27:33.990191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990196 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990209 4539 cpu_manager.go:414] "containerMap: removing container" podUID="305e93ab-f9a0-4374-80a0-8c533b7684d4" containerName="maas-api" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990216 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305e93ab-f9a0-4374-80a0-8c533b7684d4" containerName="maas-api" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990387 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c7d2f5-ee54-4e58-9f7d-48987054010f" containerName="istio-proxy" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990403 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="428f9575-cc5c-4cb1-87b8-cc851778cdea" containerName="main" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990414 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990441 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c73247-c9ce-408e-8569-d1eb2eaaa77d" containerName="maas-api" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990455 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee28d928-7be1-4218-b50d-f1fd00a37eca" containerName="istio-proxy" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990467 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3667ce2-6425-4264-b6bc-592f6272b109" containerName="main" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990483 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c791259-d616-4cd1-8def-c481def1035f" containerName="maas-api" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990491 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="144603bc-fbc4-4006-91ff-7f08ab5d1769" containerName="main" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990499 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5605eca6-fcbd-4702-b85e-54ec28aa86a2" containerName="istio-proxy" Nov 03 17:27:33.991024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.990508 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="305e93ab-f9a0-4374-80a0-8c533b7684d4" containerName="maas-api" Nov 03 17:27:33.994721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:33.994696 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.022684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.022654 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg"] Nov 03 17:27:34.088155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088111 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088154 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929514d9-b731-4a30-8a67-6d1522c756cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088434 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088197 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hljxx\" (UniqueName: \"kubernetes.io/projected/929514d9-b731-4a30-8a67-6d1522c756cc-kube-api-access-hljxx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088434 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088309 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929514d9-b731-4a30-8a67-6d1522c756cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088544 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088447 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929514d9-b731-4a30-8a67-6d1522c756cc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088544 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088504 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088641 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088573 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088767 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.088846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.088824 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190518 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190482 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929514d9-b731-4a30-8a67-6d1522c756cc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190570 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190594 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190711 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190740 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190761 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.190831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190776 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929514d9-b731-4a30-8a67-6d1522c756cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190854 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hljxx\" (UniqueName: \"kubernetes.io/projected/929514d9-b731-4a30-8a67-6d1522c756cc-kube-api-access-hljxx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.190897 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929514d9-b731-4a30-8a67-6d1522c756cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.191248 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.191547 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.191551 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929514d9-b731-4a30-8a67-6d1522c756cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.191586 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.191750 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.191692 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.193772 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.193749 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929514d9-b731-4a30-8a67-6d1522c756cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.193973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.193956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929514d9-b731-4a30-8a67-6d1522c756cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.224859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.224820 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hljxx\" (UniqueName: \"kubernetes.io/projected/929514d9-b731-4a30-8a67-6d1522c756cc-kube-api-access-hljxx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.225713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.225688 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929514d9-b731-4a30-8a67-6d1522c756cc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7dnkg\" (UID: \"929514d9-b731-4a30-8a67-6d1522c756cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.304394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.304342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:34.322374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.322339 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:34.520163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.520112 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:34.697047 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.697014 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg"] Nov 03 17:27:34.698621 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:27:34.698594 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929514d9_b731_4a30_8a67_6d1522c756cc.slice/crio-fb962e55b95fddc76a2896f900833ecd7a0e212f6b357bd28a3bc00a01d92018 WatchSource:0}: Error finding container fb962e55b95fddc76a2896f900833ecd7a0e212f6b357bd28a3bc00a01d92018: Status 404 returned error can't find the container with id fb962e55b95fddc76a2896f900833ecd7a0e212f6b357bd28a3bc00a01d92018 Nov 03 17:27:34.701894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.701852 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:27:34.702015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.701948 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:27:34.702015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.701991 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:27:34.784883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:34.784847 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:35.649014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:35.648973 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:35.741483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:35.741409 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" event={"ID":"929514d9-b731-4a30-8a67-6d1522c756cc","Type":"ContainerStarted","Data":"7537791d5c28706cc8f4c39bb8ea33c09785d890cfd57d5833d814e32130601a"} Nov 03 17:27:35.741704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:35.741489 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" event={"ID":"929514d9-b731-4a30-8a67-6d1522c756cc","Type":"ContainerStarted","Data":"fb962e55b95fddc76a2896f900833ecd7a0e212f6b357bd28a3bc00a01d92018"} Nov 03 17:27:35.818483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:35.815626 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podStartSLOduration=2.815605972 podStartE2EDuration="2.815605972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:35.810747983 +0000 UTC m=+4399.504019030" watchObservedRunningTime="2026-11-03 17:27:35.815605972 +0000 UTC m=+4399.508876995" Nov 03 17:27:35.856661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:35.856631 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:36.324450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:36.324325 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:36.324681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:36.324523 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:36.324681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:36.324570 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:36.715509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:36.715396 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:37.330470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:37.324839 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:37.330470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:37.324916 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:37.352063 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:37.352027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:38.054187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.054153 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42"] Nov 03 17:27:38.055012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.054980 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" podUID="7d6bf843-67e8-4ccf-accf-816501f126fb" containerName="maas-api" containerID="cri-o://0e7b96bc8e8806d5855309b8885f7c86b58072c31005076270cf935e5d0565f5" gracePeriod=30 Nov 03 17:27:38.162062 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.161992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:38.392532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.381553 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:38.392532 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.381619 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:38.781214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.780624 4539 generic.go:356] "Generic (PLEG): container finished" podID="7d6bf843-67e8-4ccf-accf-816501f126fb" containerID="0e7b96bc8e8806d5855309b8885f7c86b58072c31005076270cf935e5d0565f5" exitCode=0 Nov 03 17:27:38.781214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.780675 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" event={"ID":"7d6bf843-67e8-4ccf-accf-816501f126fb","Type":"ContainerDied","Data":"0e7b96bc8e8806d5855309b8885f7c86b58072c31005076270cf935e5d0565f5"} Nov 03 17:27:38.781214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.780756 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" event={"ID":"7d6bf843-67e8-4ccf-accf-816501f126fb","Type":"ContainerDied","Data":"ec4d3d265072a97c46bd3bdeffaa7079d4437dea7814be989b886ab4e98ef426"} Nov 03 17:27:38.781214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.780785 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec4d3d265072a97c46bd3bdeffaa7079d4437dea7814be989b886ab4e98ef426" Nov 03 17:27:38.793382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.793352 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:27:38.987666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.984839 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:38.991050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.990907 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\") pod \"7d6bf843-67e8-4ccf-accf-816501f126fb\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " Nov 03 17:27:38.991050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.990987 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt\" (UniqueName: \"kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt\") pod \"7d6bf843-67e8-4ccf-accf-816501f126fb\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " Nov 03 17:27:38.991324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.991273 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs\") pod \"7d6bf843-67e8-4ccf-accf-816501f126fb\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " Nov 03 17:27:38.991385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.991348 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca\") pod \"7d6bf843-67e8-4ccf-accf-816501f126fb\" (UID: \"7d6bf843-67e8-4ccf-accf-816501f126fb\") " Nov 03 17:27:38.993144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.993112 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca" pod "7d6bf843-67e8-4ccf-accf-816501f126fb" (UID: "7d6bf843-67e8-4ccf-accf-816501f126fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:38.996491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.996441 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls" pod "7d6bf843-67e8-4ccf-accf-816501f126fb" (UID: "7d6bf843-67e8-4ccf-accf-816501f126fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:27:38.997188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.997149 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt" pod "7d6bf843-67e8-4ccf-accf-816501f126fb" (UID: "7d6bf843-67e8-4ccf-accf-816501f126fb"). InnerVolumeSpecName "kube-api-access-gg5rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:38.998131 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:38.998084 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs" pod "7d6bf843-67e8-4ccf-accf-816501f126fb" (UID: "7d6bf843-67e8-4ccf-accf-816501f126fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:27:39.092821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.092782 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:39.092821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.092819 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6bf843-67e8-4ccf-accf-816501f126fb-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:39.093377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.092830 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6bf843-67e8-4ccf-accf-816501f126fb-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:39.093377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.092840 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5rt\" (UniqueName: \"kubernetes.io/projected/7d6bf843-67e8-4ccf-accf-816501f126fb-kube-api-access-gg5rt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:39.330555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.326675 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:39.330555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.326740 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:39.787408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.787371 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42" Nov 03 17:27:39.862391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.862347 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42"] Nov 03 17:27:39.872467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:39.872437 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89df9a-6b8f5448bb-2zr42"] Nov 03 17:27:40.324318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:40.323809 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:40.324318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:40.323904 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:40.470843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:40.470808 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6bf843-67e8-4ccf-accf-816501f126fb" path="/var/lib/kubelet/pods/7d6bf843-67e8-4ccf-accf-816501f126fb/volumes" Nov 03 17:27:40.724497 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:40.724333 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:41.323236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:41.323196 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:41.323480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:41.323264 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:42.256049 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:42.256015 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:42.322841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:42.322808 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 17:27:42.323116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:42.322867 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 17:27:42.404688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:42.404651 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:43.001023 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.000981 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:43.305444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.305373 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw"] Nov 03 17:27:43.306309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.306269 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" containerID="cri-o://d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526" gracePeriod=30 Nov 03 17:27:43.317386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.317352 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc"] Nov 03 17:27:43.319063 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.317636 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" containerID="cri-o://16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f" gracePeriod=30 Nov 03 17:27:43.386145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.386075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:43.900320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.900288 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:43.901021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.900991 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:43.902168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:43.902148 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" Nov 03 17:27:44.048052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:44.048003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:44.434462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:44.434393 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:45.246772 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:45.246722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:45.422522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:45.422406 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:45.885232 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:45.884679 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:46.515881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:46.515843 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:27:46.516229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:46.516189 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:27:47.372957 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:47.372920 4539 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:27:47.373484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:47.372995 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:27:47.552953 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:47.552908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:47.834360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:47.834322 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:48.629077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.629038 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:27:48.664438 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.664407 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:27:48.685591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685551 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.685787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685721 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.685787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685760 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.685923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685827 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.685977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685919 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.685977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685950 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.686098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.685988 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.686098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.686043 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.686219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.686105 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo\") pod \"103edeb3-8c23-4d58-a70c-bb4daa474873\" (UID: \"103edeb3-8c23-4d58-a70c-bb4daa474873\") " Nov 03 17:27:48.687601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.686735 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.687601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.687008 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.687601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.687078 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:48.687601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.687219 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.687601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.687318 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.691471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.691444 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:27:48.691626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.691527 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.693583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.693546 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "kube-api-access-fjpkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.694133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.694097 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token" pod "103edeb3-8c23-4d58-a70c-bb4daa474873" (UID: "103edeb3-8c23-4d58-a70c-bb4daa474873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.787501 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787455 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.787699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787540 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.787699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787574 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.787699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787619 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.787699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787683 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.787869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787702 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.787869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787775 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.787972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.787879 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.788123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788004 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.788123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788055 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs\") pod \"778356c7-9507-4aca-91ed-8ed015d90ae4\" (UID: \"778356c7-9507-4aca-91ed-8ed015d90ae4\") " Nov 03 17:27:48.788254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788201 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.788687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788664 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.788951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788926 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788957 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788973 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.788987 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjpkb\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-kube-api-access-fjpkb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789003 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789017 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789041 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789056 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103edeb3-8c23-4d58-a70c-bb4daa474873-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789072 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789083 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789087 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789095 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789132 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103edeb3-8c23-4d58-a70c-bb4daa474873-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.789399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.789303 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:48.790297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.790274 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.790559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.790537 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:27:48.791986 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.791967 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "kube-api-access-zdp8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.792122 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.792083 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy" pod "778356c7-9507-4aca-91ed-8ed015d90ae4" (UID: "778356c7-9507-4aca-91ed-8ed015d90ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.888417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.888313 4539 generic.go:356] "Generic (PLEG): container finished" podID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerID="d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526" exitCode=0 Nov 03 17:27:48.888417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.888375 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" Nov 03 17:27:48.888417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.888401 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" event={"ID":"103edeb3-8c23-4d58-a70c-bb4daa474873","Type":"ContainerDied","Data":"d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526"} Nov 03 17:27:48.888767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.888476 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw" event={"ID":"103edeb3-8c23-4d58-a70c-bb4daa474873","Type":"ContainerDied","Data":"60b3ee06768ba6bb011bdac7ce506e6f81d6d202abc538a90a568185bbf78966"} Nov 03 17:27:48.888767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.888509 4539 scope.go:122] "RemoveContainer" containerID="d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526" Nov 03 17:27:48.890133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890110 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890137 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778356c7-9507-4aca-91ed-8ed015d90ae4-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890146 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890156 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778356c7-9507-4aca-91ed-8ed015d90ae4-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890164 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdp8p\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-kube-api-access-zdp8p\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890173 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778356c7-9507-4aca-91ed-8ed015d90ae4-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890204 4539 generic.go:356] "Generic (PLEG): container finished" podID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerID="16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f" exitCode=0 Nov 03 17:27:48.890260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890242 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" Nov 03 17:27:48.890683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890286 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" event={"ID":"778356c7-9507-4aca-91ed-8ed015d90ae4","Type":"ContainerDied","Data":"16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f"} Nov 03 17:27:48.890683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.890339 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc" event={"ID":"778356c7-9507-4aca-91ed-8ed015d90ae4","Type":"ContainerDied","Data":"4e265c49d42853b1dcaeedfc332eacb93970b7b97cdc04f9fc72eac1d2441e04"} Nov 03 17:27:48.901576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.901553 4539 scope.go:122] "RemoveContainer" containerID="d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526" Nov 03 17:27:48.901943 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:27:48.901921 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526\": container with ID starting with d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526 not found: ID does not exist" containerID="d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526" Nov 03 17:27:48.902051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.901954 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526"} err="failed to get container status \"d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526\": rpc error: code = NotFound desc = could not find container \"d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526\": container with ID starting with d1cdeb4a415ec66ff0916ffd2babc1330facbf7e72fd9b56072eca7df7864526 not found: ID does not exist" Nov 03 17:27:48.902051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.901972 4539 scope.go:122] "RemoveContainer" containerID="16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f" Nov 03 17:27:48.913247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.913223 4539 scope.go:122] "RemoveContainer" containerID="16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f" Nov 03 17:27:48.913655 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:27:48.913629 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f\": container with ID starting with 16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f not found: ID does not exist" containerID="16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f" Nov 03 17:27:48.913761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.913665 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f"} err="failed to get container status \"16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f\": rpc error: code = NotFound desc = could not find container \"16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f\": container with ID starting with 16002637be2bd7d1eb2dc33a5afabca441c9f43d3d0822919b50350cb73a321f not found: ID does not exist" Nov 03 17:27:48.940008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.939972 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw"] Nov 03 17:27:48.946180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.946141 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-x8rhw"] Nov 03 17:27:48.971922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.971891 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc"] Nov 03 17:27:48.976852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:48.976821 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-tx6mc"] Nov 03 17:27:49.174275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:49.174180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:49.342493 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:49.336804 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:50.463130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:50.462382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:50.472352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:50.472317 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" path="/var/lib/kubelet/pods/103edeb3-8c23-4d58-a70c-bb4daa474873/volumes" Nov 03 17:27:50.472913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:50.472897 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" path="/var/lib/kubelet/pods/778356c7-9507-4aca-91ed-8ed015d90ae4/volumes" Nov 03 17:27:50.891294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:50.891260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:51.055225 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:51.055188 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:51.227609 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:51.227226 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:52.573083 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:52.573040 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:53.976564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:53.976533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:54.115663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:54.115630 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:55.564370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:55.564340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:56.158624 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:27:56.158584 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:27:56.926717 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:56.926680 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:58.284576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:58.284541 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:27:59.863790 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:27:59.863760 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:01.326575 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:01.326539 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:02.781513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:02.780972 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:04.293916 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:04.293817 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:05.852209 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:05.852175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:07.226864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:07.226829 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:08.697162 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:08.697114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:10.012490 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:10.012403 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:11.359039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:11.358994 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:12.766158 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:12.766123 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:14.210303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:14.210267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:15.600148 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:15.600111 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:16.895980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:16.895940 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:16.975582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:16.975548 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:28:16.975820 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:16.975613 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zvhjn" podUID="74c46d2b-77a9-4239-a40c-2b2152f04e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:28:17.351284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:17.351251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:18.677262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:18.677230 4539 scope.go:122] "RemoveContainer" containerID="0e7b96bc8e8806d5855309b8885f7c86b58072c31005076270cf935e5d0565f5" Nov 03 17:28:18.750208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:18.750148 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:20.216814 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:20.216768 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:21.684999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:21.684761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:23.050984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:23.050950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:24.506364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:24.506323 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:25.995551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:25.990216 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:26.159723 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:28:26.159674 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:28:27.432905 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:27.432866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:27.593351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:27.593320 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:28.958168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:28.957895 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:30.467021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:30.466991 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:32.042100 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:32.042070 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:33.540623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:33.540582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:34.128787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.128755 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5"] Nov 03 17:28:34.129471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129451 4539 cpu_manager.go:414] "containerMap: removing container" podUID="7d6bf843-67e8-4ccf-accf-816501f126fb" containerName="maas-api" Nov 03 17:28:34.129471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129471 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6bf843-67e8-4ccf-accf-816501f126fb" containerName="maas-api" Nov 03 17:28:34.129675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129534 4539 cpu_manager.go:414] "containerMap: removing container" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" Nov 03 17:28:34.129675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129543 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" Nov 03 17:28:34.129675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129560 4539 cpu_manager.go:414] "containerMap: removing container" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" Nov 03 17:28:34.129675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129567 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" Nov 03 17:28:34.129879 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129752 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6bf843-67e8-4ccf-accf-816501f126fb" containerName="maas-api" Nov 03 17:28:34.129879 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129778 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="103edeb3-8c23-4d58-a70c-bb4daa474873" containerName="istio-proxy" Nov 03 17:28:34.129879 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.129789 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="778356c7-9507-4aca-91ed-8ed015d90ae4" containerName="istio-proxy" Nov 03 17:28:34.133244 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.133220 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.173946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.173910 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5"] Nov 03 17:28:34.282461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282387 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282536 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nts5t\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282563 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282647 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282687 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282726 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282769 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282802 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.282919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.282910 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384022 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nts5t\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384124 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384070 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384127 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384156 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384214 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384255 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.384385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384286 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.385142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384464 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.385142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.384565 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.385142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.385063 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.386743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.386686 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.387155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.387133 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.388325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.388242 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.388500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.388389 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.389250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.389225 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.390075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.390054 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.409006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.408969 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nts5t\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.411123 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.411096 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wlxs5\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.448630 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.448591 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:34.781573 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.781545 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5"] Nov 03 17:28:34.782577 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:28:34.782547 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5929f511_5e10_44ff_94d3_68c7d7a6e6cb.slice/crio-f69ffa1a02cb98b7e367906e6a8e5fff5b056d696f17647599d076315b64a411 WatchSource:0}: Error finding container f69ffa1a02cb98b7e367906e6a8e5fff5b056d696f17647599d076315b64a411: Status 404 returned error can't find the container with id f69ffa1a02cb98b7e367906e6a8e5fff5b056d696f17647599d076315b64a411 Nov 03 17:28:34.785941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.785904 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:28:34.786047 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.785976 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:28:34.786047 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.786006 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:28:34.910647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:34.910607 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:35.290294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:35.290258 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" event={"ID":"5929f511-5e10-44ff-94d3-68c7d7a6e6cb","Type":"ContainerStarted","Data":"9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88"} Nov 03 17:28:35.290544 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:35.290313 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" event={"ID":"5929f511-5e10-44ff-94d3-68c7d7a6e6cb","Type":"ContainerStarted","Data":"f69ffa1a02cb98b7e367906e6a8e5fff5b056d696f17647599d076315b64a411"} Nov 03 17:28:35.352401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:35.352330 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podStartSLOduration=1.352306954 podStartE2EDuration="1.352306954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:35.347729178 +0000 UTC m=+4459.041000202" watchObservedRunningTime="2026-11-03 17:28:35.352306954 +0000 UTC m=+4459.045577976" Nov 03 17:28:35.450786 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:35.450748 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:35.453384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:35.452898 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:35.453384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:35.452972 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:36.396227 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:36.396185 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:36.455462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:36.451790 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:36.455462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:36.451845 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:37.449414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:37.449381 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:37.449865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:37.449488 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:37.866995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:37.866961 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:38.450065 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:38.450016 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:38.450517 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:38.450100 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:39.360511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:39.360457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:39.449161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:39.449121 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:39.449351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:39.449201 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:40.449858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:40.449818 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:40.450313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:40.449910 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:41.048835 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:41.048791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:41.449888 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:41.449789 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:41.449888 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:41.449864 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:42.449085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:42.449046 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 03 17:28:42.449287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:42.449098 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 03 17:28:42.498189 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:42.498149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:43.611743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:43.611710 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:43.612802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:43.612770 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:43.613881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:43.613861 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:28:43.973007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:43.972922 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:45.471340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:45.471303 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:47.028474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:47.028416 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:48.352502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:48.352469 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:48.512957 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:48.512919 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:48.840503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:48.840461 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:50.109491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:50.109412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:51.708451 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:51.708405 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:53.284700 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:53.284607 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:54.813523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:54.813480 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:55.180208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:55.180104 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:55.344063 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:55.344029 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:56.169901 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:28:56.169861 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:28:56.362828 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:56.362789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:57.117486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.117442 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79"] Nov 03 17:28:57.122011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.121977 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.149631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.149596 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a9426b\"/\"kube-root-ca.crt\"" Nov 03 17:28:57.149819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.149775 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a9426b\"/\"openshift-service-ca.crt\"" Nov 03 17:28:57.149978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.149960 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a9426b\"/\"sub-test-model-a9426b-kserve-self-signed-certs\"" Nov 03 17:28:57.150052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.150040 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a9426b\"/\"default-dockercfg-vt6zf\"" Nov 03 17:28:57.156635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.156605 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79"] Nov 03 17:28:57.162063 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.162030 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.163657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.163625 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.163771 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.163718 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.163827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.163786 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.163900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.163886 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.164011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.163995 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfngp\" (UniqueName: \"kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.249259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.249220 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:57.264941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.264898 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.264996 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265079 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfngp\" (UniqueName: \"kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265147 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265369 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265247 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265369 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265311 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265517 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265497 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265718 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.265944 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.265915 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.268000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.267977 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.268130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.268111 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.289560 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.289532 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfngp\" (UniqueName: \"kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp\") pod \"sub-test-model-a9426b-kserve-84fd685b85-k6d79\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.470999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.470892 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:57.507265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.507169 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:57.710518 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.703362 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:57.862276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.862242 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:57.901687 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:28:57.901643 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56322f04_5883_4048_bac9_338165b17c62.slice/crio-4c2259c97e535ab5e5aa2b27c48a1781a306a0a9660410e88c6c6981320e289c WatchSource:0}: Error finding container 4c2259c97e535ab5e5aa2b27c48a1781a306a0a9660410e88c6c6981320e289c: Status 404 returned error can't find the container with id 4c2259c97e535ab5e5aa2b27c48a1781a306a0a9660410e88c6c6981320e289c Nov 03 17:28:57.902475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:57.902446 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79"] Nov 03 17:28:58.163907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:58.163790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:28:58.523933 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:58.523892 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" event={"ID":"56322f04-5883-4048-bac9-338165b17c62","Type":"ContainerStarted","Data":"513a403f7256170a513c21a2a04c6b65c44dfc42c509b335b9c6ff2c40394e61"} Nov 03 17:28:58.524597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:58.523947 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" event={"ID":"56322f04-5883-4048-bac9-338165b17c62","Type":"ContainerStarted","Data":"4c2259c97e535ab5e5aa2b27c48a1781a306a0a9660410e88c6c6981320e289c"} Nov 03 17:28:58.576693 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:58.576639 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" podStartSLOduration=1.576623909 podStartE2EDuration="1.576623909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:58.57585489 +0000 UTC m=+4482.269125974" watchObservedRunningTime="2026-11-03 17:28:58.576623909 +0000 UTC m=+4482.269894930" Nov 03 17:28:59.471219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:59.471175 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:28:59.650643 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:28:59.650608 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:01.218815 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:01.218781 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:02.815759 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:02.815716 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:04.257247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:04.257209 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:04.501025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:04.500989 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:04.658235 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:04.658198 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:04.988971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:04.988859 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:05.733206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:05.733168 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:07.267339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:07.267300 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:07.471277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:07.471241 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:29:07.484628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:07.484595 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:29:07.604560 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:07.604528 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:29:08.656527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:08.656477 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:10.208215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:10.208182 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:10.960675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:10.960639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:11.103319 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:11.103286 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:11.572345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:11.572303 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:11.723394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:11.723359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:11.917375 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:11.917238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:12.165337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:12.165292 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:13.510313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:13.510144 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:15.181610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:15.181566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:16.758665 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:16.758631 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:17.039695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:17.039665 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:17.369280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:17.369183 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:29:17.391294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:17.391243 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:29:18.239241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:18.239146 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:18.394510 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:18.394475 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:19.050719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:19.050689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:20.029292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:20.020982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:21.573741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:21.573711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:23.165298 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.165255 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:23.716799 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.716760 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk"] Nov 03 17:29:23.728754 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.728719 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.754919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.754887 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk"] Nov 03 17:29:23.762166 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.762139 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e2d835\"/\"default-dockercfg-zxsn7\"" Nov 03 17:29:23.762354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.762145 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e2d835\"/\"kube-root-ca.crt\"" Nov 03 17:29:23.762354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.762157 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e2d835\"/\"sub-test-model-e2d835-kserve-self-signed-certs\"" Nov 03 17:29:23.762354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.762173 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e2d835\"/\"openshift-service-ca.crt\"" Nov 03 17:29:23.786461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.786380 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vzgs\" (UniqueName: \"kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.786663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.786487 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.786663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.786607 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.786663 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.786655 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.786864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.786746 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.786864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.786799 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.818511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.818482 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:23.888193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888162 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vzgs\" (UniqueName: \"kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888197 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888436 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888249 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888436 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888278 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888436 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888314 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888695 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888813 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888792 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.888871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.888807 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.889095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.889070 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.891121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.891095 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.891522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.891503 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:23.914882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:23.914848 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vzgs\" (UniqueName: \"kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs\") pod \"sub-test-model-e2d835-kserve-c6b65d759-5g6hk\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:24.061446 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:24.060682 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:24.394219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:24.394133 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk"] Nov 03 17:29:24.394648 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:29:24.394510 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb3c920_1180_4861_aaa1_87df4bdc1516.slice/crio-689ecdff2bb5fe6d3c3d53e2c54d9f89d634521d2094786fd57d1535bc95f0c4 WatchSource:0}: Error finding container 689ecdff2bb5fe6d3c3d53e2c54d9f89d634521d2094786fd57d1535bc95f0c4: Status 404 returned error can't find the container with id 689ecdff2bb5fe6d3c3d53e2c54d9f89d634521d2094786fd57d1535bc95f0c4 Nov 03 17:29:24.743384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:24.743286 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" event={"ID":"1fb3c920-1180-4861-aaa1-87df4bdc1516","Type":"ContainerStarted","Data":"4ad95ca90a234a2b2f88b0c5bed748a3ee24f956c22fe012f98f0f35249c4188"} Nov 03 17:29:24.743384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:24.743333 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" event={"ID":"1fb3c920-1180-4861-aaa1-87df4bdc1516","Type":"ContainerStarted","Data":"689ecdff2bb5fe6d3c3d53e2c54d9f89d634521d2094786fd57d1535bc95f0c4"} Nov 03 17:29:24.789530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:24.789461 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" podStartSLOduration=1.789416357 podStartE2EDuration="1.789416357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:24.785883174 +0000 UTC m=+4508.479154196" watchObservedRunningTime="2026-11-03 17:29:24.789416357 +0000 UTC m=+4508.482687379" Nov 03 17:29:25.013159 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:25.013062 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:25.061740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:25.061698 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:26.160161 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:29:26.160122 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:29:26.586390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:26.586357 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:28.196756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:28.196724 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:29.037785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:29.037749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:29.239815 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:29.239771 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:30.930473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:30.927327 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:32.559167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:32.559126 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:34.061708 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:34.061666 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:34.074901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:34.074873 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:34.111934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:34.111902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:34.275775 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:34.275740 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:34.819701 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:34.819664 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:34.871761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:34.871730 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:29:35.241280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:35.241193 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz"] Nov 03 17:29:35.241836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:35.241475 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" containerID="cri-o://83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8" gracePeriod=30 Nov 03 17:29:35.294500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:35.294465 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:35.964911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:35.964873 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:36.165806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:36.165763 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:36.336011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:36.335974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:37.594460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:37.594401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:39.174187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:39.174146 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:40.532711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.532682 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:29:40.656258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656215 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656316 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656354 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656394 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656448 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656512 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656656 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656740 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656695 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656729 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.656895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656749 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo\") pod \"b59c0609-bbd8-4872-9a9c-071780f69931\" (UID: \"b59c0609-bbd8-4872-9a9c-071780f69931\") " Nov 03 17:29:40.656895 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656724 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:29:40.657075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.656985 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.657075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657061 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.657292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657271 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.657557 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657538 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.657679 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657663 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.657794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657781 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.657892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657880 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59c0609-bbd8-4872-9a9c-071780f69931-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.658004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.657975 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.659841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.659811 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:29:40.660768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.660724 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "kube-api-access-xflvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:40.661853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.661043 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.661853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.661044 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token" pod "b59c0609-bbd8-4872-9a9c-071780f69931" (UID: "b59c0609-bbd8-4872-9a9c-071780f69931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:40.759789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.759750 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59c0609-bbd8-4872-9a9c-071780f69931-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.759789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.759785 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.759789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.759796 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59c0609-bbd8-4872-9a9c-071780f69931-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.760052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.759804 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflvp\" (UniqueName: \"kubernetes.io/projected/b59c0609-bbd8-4872-9a9c-071780f69931-kube-api-access-xflvp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.836464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.836392 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:40.931017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.930977 4539 generic.go:356] "Generic (PLEG): container finished" podID="b59c0609-bbd8-4872-9a9c-071780f69931" containerID="83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8" exitCode=0 Nov 03 17:29:40.931251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.931053 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" Nov 03 17:29:40.931251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.931086 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" event={"ID":"b59c0609-bbd8-4872-9a9c-071780f69931","Type":"ContainerDied","Data":"83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8"} Nov 03 17:29:40.931251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.931214 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz" event={"ID":"b59c0609-bbd8-4872-9a9c-071780f69931","Type":"ContainerDied","Data":"d615ba5cbec3d11e30d6c1f88192ae92f2b0ed3b77d3d04d131f45dfd683f670"} Nov 03 17:29:40.931251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.931248 4539 scope.go:122] "RemoveContainer" containerID="83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8" Nov 03 17:29:40.947325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.947297 4539 scope.go:122] "RemoveContainer" containerID="83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8" Nov 03 17:29:40.947759 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:29:40.947732 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8\": container with ID starting with 83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8 not found: ID does not exist" containerID="83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8" Nov 03 17:29:40.947854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.947777 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8"} err="failed to get container status \"83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8\": rpc error: code = NotFound desc = could not find container \"83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8\": container with ID starting with 83bc801fe278d22f180e0dcbbdee3bc4473f91e3d1d4c1304e15c6042829fad8 not found: ID does not exist" Nov 03 17:29:40.985557 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.985514 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz"] Nov 03 17:29:40.989733 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:40.989703 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-qqmkz"] Nov 03 17:29:42.394196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:42.394167 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:42.470331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:42.470301 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" path="/var/lib/kubelet/pods/b59c0609-bbd8-4872-9a9c-071780f69931/volumes" Nov 03 17:29:42.825462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:42.825407 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:43.041043 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:43.041009 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:43.193610 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:43.193529 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:44.169673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:44.169624 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:45.777067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:45.777028 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:47.514198 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:47.514159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:48.675685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:48.675652 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:48.964916 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:48.964821 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:50.719877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:50.719848 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:52.422701 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:52.422662 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:54.086517 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:54.086475 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:55.880743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:55.880699 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:56.171936 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:29:56.171850 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:29:57.550874 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:57.550836 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:59.057536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:59.057459 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:59.804434 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:59.804377 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:29:59.955962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:29:59.955885 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:00.303965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.303918 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v"] Nov 03 17:30:00.305760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.305725 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" Nov 03 17:30:00.305760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.305760 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" Nov 03 17:30:00.306032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.306015 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59c0609-bbd8-4872-9a9c-071780f69931" containerName="istio-proxy" Nov 03 17:30:00.310295 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.310274 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.336108 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.336072 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v"] Nov 03 17:30:00.389674 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.389635 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:30:00.471159 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.471124 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-6zf4v\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.471343 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.471314 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwf4\" (UniqueName: \"kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4\") pod \"maas-api-key-cleanup-29895450-6zf4v\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.572702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.572606 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwf4\" (UniqueName: \"kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4\") pod \"maas-api-key-cleanup-29895450-6zf4v\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.572907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.572743 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-6zf4v\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.573396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.573376 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895450-6zf4v\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.600872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.600827 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwf4\" (UniqueName: \"kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4\") pod \"maas-api-key-cleanup-29895450-6zf4v\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.634709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.634672 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:00.746999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.746954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:00.949962 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:30:00.949927 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb776568_4552_40bd_9238_288382562539.slice/crio-051f99e75871d3305c58f9128f29737764000947e4cbd1f2f900f57d475e7432 WatchSource:0}: Error finding container 051f99e75871d3305c58f9128f29737764000947e4cbd1f2f900f57d475e7432: Status 404 returned error can't find the container with id 051f99e75871d3305c58f9128f29737764000947e4cbd1f2f900f57d475e7432 Nov 03 17:30:00.950347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:00.950324 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v"] Nov 03 17:30:01.142175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:01.142125 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerStarted","Data":"051f99e75871d3305c58f9128f29737764000947e4cbd1f2f900f57d475e7432"} Nov 03 17:30:01.653631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:01.653542 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:02.152418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:02.152376 4539 generic.go:356] "Generic (PLEG): container finished" podID="cb776568-4552-40bd-9238-288382562539" containerID="ac5a38d123a8ab0fbdf3c88957678e767470003e1969f3b37ec887827b4aa174" exitCode=60 Nov 03 17:30:02.152736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:02.152455 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerDied","Data":"ac5a38d123a8ab0fbdf3c88957678e767470003e1969f3b37ec887827b4aa174"} Nov 03 17:30:02.152854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:02.152835 4539 scope.go:122] "RemoveContainer" containerID="ac5a38d123a8ab0fbdf3c88957678e767470003e1969f3b37ec887827b4aa174" Nov 03 17:30:02.452418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:02.452330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:03.162302 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:03.162251 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerStarted","Data":"9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c"} Nov 03 17:30:03.203785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:03.203725 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" podStartSLOduration=3.203706434 podStartE2EDuration="3.203706434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.202390816 +0000 UTC m=+4546.895661838" watchObservedRunningTime="2026-11-03 17:30:03.203706434 +0000 UTC m=+4546.896977457" Nov 03 17:30:04.074730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:04.074689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:04.173097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:04.173064 4539 generic.go:356] "Generic (PLEG): container finished" podID="cb776568-4552-40bd-9238-288382562539" containerID="9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c" exitCode=60 Nov 03 17:30:04.173631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:04.173147 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerDied","Data":"9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c"} Nov 03 17:30:04.173631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:04.173208 4539 scope.go:122] "RemoveContainer" containerID="ac5a38d123a8ab0fbdf3c88957678e767470003e1969f3b37ec887827b4aa174" Nov 03 17:30:04.173631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:04.173602 4539 scope.go:122] "RemoveContainer" containerID="9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c" Nov 03 17:30:04.174045 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:04.174019 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-6zf4v_odh-ai-gateway-infra(cb776568-4552-40bd-9238-288382562539)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" podUID="cb776568-4552-40bd-9238-288382562539" Nov 03 17:30:05.182809 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:05.182785 4539 scope.go:122] "RemoveContainer" containerID="9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c" Nov 03 17:30:05.183236 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:05.183144 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-6zf4v_odh-ai-gateway-infra(cb776568-4552-40bd-9238-288382562539)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" podUID="cb776568-4552-40bd-9238-288382562539" Nov 03 17:30:05.646018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:05.645939 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:07.206408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:07.206372 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq"] Nov 03 17:30:07.241624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:07.241593 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc"] Nov 03 17:30:07.475805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:07.475707 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:07.629414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:07.629380 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:09.272008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:09.271969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:09.688318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:09.688278 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:10.829025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:10.828982 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:12.207368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:12.207312 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerName="payload-processing" containerID="cri-o://abbd64f7b09dfa7cf8286782fd77db38ee54a2c7c41f4691d8afd3589deed2f4" gracePeriod=25 Nov 03 17:30:12.242457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:12.242396 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerName="payload-pre-processing" containerID="cri-o://efa876c4866f64006666c3ec90927ba8f97a150a56be41e5269307637ce2bd48" gracePeriod=25 Nov 03 17:30:12.415852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:12.415815 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:13.970051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:13.970016 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:15.559371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:15.559284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:16.470174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:16.470139 4539 scope.go:122] "RemoveContainer" containerID="9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c" Nov 03 17:30:16.712301 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:16.712266 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:17.273920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:17.273884 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerStarted","Data":"809187a546cb7fc257910214915b2d5d27473ea12bc0c8d37227a031d9c17df0"} Nov 03 17:30:17.301708 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:17.301663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:17.449870 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:17.449837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:18.283154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:18.283121 4539 generic.go:356] "Generic (PLEG): container finished" podID="cb776568-4552-40bd-9238-288382562539" containerID="809187a546cb7fc257910214915b2d5d27473ea12bc0c8d37227a031d9c17df0" exitCode=60 Nov 03 17:30:18.283651 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:18.283194 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerDied","Data":"809187a546cb7fc257910214915b2d5d27473ea12bc0c8d37227a031d9c17df0"} Nov 03 17:30:18.283651 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:18.283255 4539 scope.go:122] "RemoveContainer" containerID="9a65cfa196d9fd015f53650b715f667ee631e7f6d53752b7a4bf87d2524e227c" Nov 03 17:30:18.283789 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:18.283727 4539 scope.go:122] "RemoveContainer" containerID="809187a546cb7fc257910214915b2d5d27473ea12bc0c8d37227a031d9c17df0" Nov 03 17:30:18.284066 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:18.284042 4539 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895450-6zf4v_odh-ai-gateway-infra(cb776568-4552-40bd-9238-288382562539)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" podUID="cb776568-4552-40bd-9238-288382562539" Nov 03 17:30:18.344135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:18.344094 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v"] Nov 03 17:30:18.900888 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:18.900851 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:19.478239 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.478203 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:19.668849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.668757 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca\") pod \"cb776568-4552-40bd-9238-288382562539\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " Nov 03 17:30:19.669144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.669047 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4\" (UniqueName: \"kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4\") pod \"cb776568-4552-40bd-9238-288382562539\" (UID: \"cb776568-4552-40bd-9238-288382562539\") " Nov 03 17:30:19.669227 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.669141 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca" pod "cb776568-4552-40bd-9238-288382562539" (UID: "cb776568-4552-40bd-9238-288382562539"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:19.669594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.669574 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb776568-4552-40bd-9238-288382562539-maas-api-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:19.671407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.671386 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4" pod "cb776568-4552-40bd-9238-288382562539" (UID: "cb776568-4552-40bd-9238-288382562539"). InnerVolumeSpecName "kube-api-access-svwf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:19.770884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:19.770853 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwf4\" (UniqueName: \"kubernetes.io/projected/cb776568-4552-40bd-9238-288382562539-kube-api-access-svwf4\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:20.250336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.250301 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6"] Nov 03 17:30:20.250736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.250575 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" podUID="9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" containerName="maas-api" containerID="cri-o://a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a" gracePeriod=30 Nov 03 17:30:20.304472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.302161 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" event={"ID":"cb776568-4552-40bd-9238-288382562539","Type":"ContainerDied","Data":"051f99e75871d3305c58f9128f29737764000947e4cbd1f2f900f57d475e7432"} Nov 03 17:30:20.304472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.302228 4539 scope.go:122] "RemoveContainer" containerID="809187a546cb7fc257910214915b2d5d27473ea12bc0c8d37227a031d9c17df0" Nov 03 17:30:20.304472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.302340 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v" Nov 03 17:30:20.369696 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.369660 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:20.375917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.375852 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v"] Nov 03 17:30:20.381270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.381240 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895450-6zf4v"] Nov 03 17:30:20.471629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.471596 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb776568-4552-40bd-9238-288382562539" path="/var/lib/kubelet/pods/cb776568-4552-40bd-9238-288382562539/volumes" Nov 03 17:30:20.830564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.830535 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:30:20.904400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.904359 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca\") pod \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " Nov 03 17:30:20.904764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.904740 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls\") pod \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " Nov 03 17:30:20.904866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.904796 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx\" (UniqueName: \"kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx\") pod \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " Nov 03 17:30:20.904866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.904837 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs\") pod \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\" (UID: \"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375\") " Nov 03 17:30:20.904866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.904843 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca" pod "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" (UID: "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:20.905345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.905322 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:20.908070 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.908032 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls" pod "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" (UID: "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:20.908182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.908045 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs" pod "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" (UID: "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:20.908182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:20.908150 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx" pod "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" (UID: "9fbf26c4-2f14-46ad-bd7e-c019d3e4d375"). InnerVolumeSpecName "kube-api-access-q72hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.006873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.006834 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.006873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.006867 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72hx\" (UniqueName: \"kubernetes.io/projected/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-kube-api-access-q72hx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.006873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.006879 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.078189 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.078098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:21.311990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.311950 4539 generic.go:356] "Generic (PLEG): container finished" podID="9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" containerID="a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a" exitCode=0 Nov 03 17:30:21.312216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.312014 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" Nov 03 17:30:21.312216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.312037 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" event={"ID":"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375","Type":"ContainerDied","Data":"a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a"} Nov 03 17:30:21.312216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.312104 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6" event={"ID":"9fbf26c4-2f14-46ad-bd7e-c019d3e4d375","Type":"ContainerDied","Data":"5bc231e58aada3e55ff1d8fb8beac4cec51d5187568b639bd3aef7adf5992cea"} Nov 03 17:30:21.312216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.312136 4539 scope.go:122] "RemoveContainer" containerID="a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a" Nov 03 17:30:21.326694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.326669 4539 scope.go:122] "RemoveContainer" containerID="a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a" Nov 03 17:30:21.327016 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:21.326997 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a\": container with ID starting with a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a not found: ID does not exist" containerID="a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a" Nov 03 17:30:21.327100 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.327039 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a"} err="failed to get container status \"a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a\": rpc error: code = NotFound desc = could not find container \"a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a\": container with ID starting with a42724d6f96555c7fae3bac59ff48d2ee0c82c8525078c4e29f0fd232f52284a not found: ID does not exist" Nov 03 17:30:21.367768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.367696 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6"] Nov 03 17:30:21.379708 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.379681 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fdbf0c-85cb4b966b-vk9n6"] Nov 03 17:30:21.974981 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.974947 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:21.975411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:21.974938 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:22.470044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:22.470008 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" path="/var/lib/kubelet/pods/9fbf26c4-2f14-46ad-bd7e-c019d3e4d375/volumes" Nov 03 17:30:22.739400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:22.739299 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:24.484189 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:24.484141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:25.052897 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:25.052865 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:25.209919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:25.209884 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:25.805475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:25.805442 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:26.209287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:26.207413 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:26.210921 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:26.210885 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:30:26.467786 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:26.467664 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:27.087224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:27.087177 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:28.205238 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:28.204284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:29.394623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.394588 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz"] Nov 03 17:30:29.395139 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.394841 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" podUID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" containerName="maas-api" containerID="cri-o://7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034" gracePeriod=30 Nov 03 17:30:29.426821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.426789 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" podUID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.163:8443/health\": dial tcp 10.133.0.163:8443: connect: connection refused" Nov 03 17:30:29.492058 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.492029 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:29.977929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.977905 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:30:29.987107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.987080 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls\") pod \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " Nov 03 17:30:29.987251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.987231 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca\") pod \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " Nov 03 17:30:29.987487 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.987463 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn\" (UniqueName: \"kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn\") pod \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " Nov 03 17:30:29.987578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.987538 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs\") pod \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\" (UID: \"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a\") " Nov 03 17:30:29.987636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.987572 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca" pod "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" (UID: "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:29.987994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.987977 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:29.989640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.989612 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls" pod "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" (UID: "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:29.989726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.989650 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn" pod "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" (UID: "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a"). InnerVolumeSpecName "kube-api-access-xf9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:29.989726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:29.989687 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs" pod "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" (UID: "b727cdc6-4114-46ef-be3a-fd5e4cd6a67a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:30.089739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.089700 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9nn\" (UniqueName: \"kubernetes.io/projected/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-kube-api-access-xf9nn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:30.089739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.089735 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:30.089739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.089746 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:30.198127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.197330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:30.425564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.425532 4539 generic.go:356] "Generic (PLEG): container finished" podID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" containerID="7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034" exitCode=0 Nov 03 17:30:30.426085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.425588 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" event={"ID":"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a","Type":"ContainerDied","Data":"7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034"} Nov 03 17:30:30.426085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.425602 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" Nov 03 17:30:30.426085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.425633 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz" event={"ID":"b727cdc6-4114-46ef-be3a-fd5e4cd6a67a","Type":"ContainerDied","Data":"3b0d4fc2ae31304e99a3ef533fa06044b2e91e486a039ac1fdb1c8338783c5be"} Nov 03 17:30:30.426085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.425654 4539 scope.go:122] "RemoveContainer" containerID="7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034" Nov 03 17:30:30.437371 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.437349 4539 scope.go:122] "RemoveContainer" containerID="7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034" Nov 03 17:30:30.437663 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:30.437639 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034\": container with ID starting with 7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034 not found: ID does not exist" containerID="7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034" Nov 03 17:30:30.437744 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.437676 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034"} err="failed to get container status \"7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034\": rpc error: code = NotFound desc = could not find container \"7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034\": container with ID starting with 7d6ad53e03c62c3ab14fdad3beb5e761f9b178b7486c8f42a70140455b043034 not found: ID does not exist" Nov 03 17:30:30.478474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.478365 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz"] Nov 03 17:30:30.490480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:30.490446 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2492c4-654797c7b4-6slnz"] Nov 03 17:30:31.760869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:31.760835 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:31.975187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:31.975153 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:31.975348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:31.975156 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:32.472859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.472769 4539 generic.go:356] "Generic (PLEG): container finished" podID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerID="abbd64f7b09dfa7cf8286782fd77db38ee54a2c7c41f4691d8afd3589deed2f4" exitCode=0 Nov 03 17:30:32.475237 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.475209 4539 generic.go:356] "Generic (PLEG): container finished" podID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerID="efa876c4866f64006666c3ec90927ba8f97a150a56be41e5269307637ce2bd48" exitCode=0 Nov 03 17:30:32.482216 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.482187 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" path="/var/lib/kubelet/pods/b727cdc6-4114-46ef-be3a-fd5e4cd6a67a/volumes" Nov 03 17:30:32.482633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.482610 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" event={"ID":"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d","Type":"ContainerDied","Data":"abbd64f7b09dfa7cf8286782fd77db38ee54a2c7c41f4691d8afd3589deed2f4"} Nov 03 17:30:32.482697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.482668 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" event={"ID":"c8f0db2d-138f-467c-9981-42b8a42ed55c","Type":"ContainerDied","Data":"efa876c4866f64006666c3ec90927ba8f97a150a56be41e5269307637ce2bd48"} Nov 03 17:30:32.564934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.564902 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:30:32.610674 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.610640 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:30:32.640280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.638889 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume\") pod \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " Nov 03 17:30:32.640280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.638971 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx\" (UniqueName: \"kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx\") pod \"c8f0db2d-138f-467c-9981-42b8a42ed55c\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " Nov 03 17:30:32.640280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.639038 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume\") pod \"c8f0db2d-138f-467c-9981-42b8a42ed55c\" (UID: \"c8f0db2d-138f-467c-9981-42b8a42ed55c\") " Nov 03 17:30:32.640280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.639303 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n\" (UniqueName: \"kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n\") pod \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\" (UID: \"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d\") " Nov 03 17:30:32.640694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.640640 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume" pod "589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" (UID: "589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:32.641074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.641002 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume" pod "c8f0db2d-138f-467c-9981-42b8a42ed55c" (UID: "c8f0db2d-138f-467c-9981-42b8a42ed55c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:32.648295 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.648256 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n" pod "589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" (UID: "589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d"). InnerVolumeSpecName "kube-api-access-cbj4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:32.648855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.648825 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx" pod "c8f0db2d-138f-467c-9981-42b8a42ed55c" (UID: "c8f0db2d-138f-467c-9981-42b8a42ed55c"). InnerVolumeSpecName "kube-api-access-ltpmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:32.728374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.728281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:32.741488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.741453 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:32.741488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.741483 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpmx\" (UniqueName: \"kubernetes.io/projected/c8f0db2d-138f-467c-9981-42b8a42ed55c-kube-api-access-ltpmx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:32.741488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.741493 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f0db2d-138f-467c-9981-42b8a42ed55c-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:32.741732 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.741502 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbj4n\" (UniqueName: \"kubernetes.io/projected/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d-kube-api-access-cbj4n\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:32.876609 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:32.876576 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:33.483509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.483407 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" Nov 03 17:30:33.483509 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.483466 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc" event={"ID":"c8f0db2d-138f-467c-9981-42b8a42ed55c","Type":"ContainerDied","Data":"da43bb4266150c5a1ab8d27101a46fb2a17e705612d58ddbb0e14af4284af6e9"} Nov 03 17:30:33.483820 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.483530 4539 scope.go:122] "RemoveContainer" containerID="efa876c4866f64006666c3ec90927ba8f97a150a56be41e5269307637ce2bd48" Nov 03 17:30:33.488472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.485798 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" Nov 03 17:30:33.488472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.485833 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq" event={"ID":"589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d","Type":"ContainerDied","Data":"02ba263a7171faf49d67e02409b25e02edfbf0ececc407e6ad00545a348c43c0"} Nov 03 17:30:33.495766 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.495747 4539 scope.go:122] "RemoveContainer" containerID="abbd64f7b09dfa7cf8286782fd77db38ee54a2c7c41f4691d8afd3589deed2f4" Nov 03 17:30:33.536414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.536373 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq"] Nov 03 17:30:33.540917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.540890 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2492c4-759976855f-9gzmq"] Nov 03 17:30:33.575824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.575785 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc"] Nov 03 17:30:33.579619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.579590 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2492c4-5c866bc79-s69nc"] Nov 03 17:30:33.766511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.764210 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:33.908803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:33.908765 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:34.470260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:34.470228 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" path="/var/lib/kubelet/pods/589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d/volumes" Nov 03 17:30:34.470696 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:34.470681 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" path="/var/lib/kubelet/pods/c8f0db2d-138f-467c-9981-42b8a42ed55c/volumes" Nov 03 17:30:34.487702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:34.487663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:34.639024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:34.638990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:35.256135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:35.256102 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:35.486453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:35.484750 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:35.510571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:35.508357 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb"] Nov 03 17:30:35.510571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:35.510095 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" containerID="cri-o://8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd" gracePeriod=30 Nov 03 17:30:35.642615 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:35.642582 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:36.330988 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:36.330954 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:37.177867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:37.176897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:38.663683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:38.663649 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:39.670730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:39.670696 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:40.210440 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.210389 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:40.803222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.803195 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:30:40.972774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.972742 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.972978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.972866 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.972978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.972910 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.972978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.972933 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.972978 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.972969 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.973224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973013 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.973224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973124 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.973224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973183 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.973224 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973209 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy\") pod \"80150068-7b02-4d59-84f7-1e23530dd995\" (UID: \"80150068-7b02-4d59-84f7-1e23530dd995\") " Nov 03 17:30:40.973486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973349 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:40.973558 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973480 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:40.973623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973567 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:40.973623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973590 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:40.973773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973751 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:40.973955 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973936 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:40.974014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973963 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80150068-7b02-4d59-84f7-1e23530dd995-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:40.974014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973978 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:40.974014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.973996 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:40.974014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.974013 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:40.975499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.975476 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "kube-api-access-xv5hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:40.975686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.975667 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:30:40.976039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.976021 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:40.976107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:40.976020 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy" pod "80150068-7b02-4d59-84f7-1e23530dd995" (UID: "80150068-7b02-4d59-84f7-1e23530dd995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:41.075253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.075172 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5hh\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-kube-api-access-xv5hh\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:41.075253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.075199 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80150068-7b02-4d59-84f7-1e23530dd995-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:41.075253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.075209 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80150068-7b02-4d59-84f7-1e23530dd995-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:41.075253 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.075217 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80150068-7b02-4d59-84f7-1e23530dd995-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:41.563983 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.563948 4539 generic.go:356] "Generic (PLEG): container finished" podID="80150068-7b02-4d59-84f7-1e23530dd995" containerID="8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd" exitCode=0 Nov 03 17:30:41.564178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.564005 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" Nov 03 17:30:41.564178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.564037 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" event={"ID":"80150068-7b02-4d59-84f7-1e23530dd995","Type":"ContainerDied","Data":"8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd"} Nov 03 17:30:41.564178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.564096 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb" event={"ID":"80150068-7b02-4d59-84f7-1e23530dd995","Type":"ContainerDied","Data":"ef848f0c19649eaab91cc463088bd58c71b88d4ddc61f03441de838f13c84381"} Nov 03 17:30:41.564178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.564129 4539 scope.go:122] "RemoveContainer" containerID="8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd" Nov 03 17:30:41.576281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.576263 4539 scope.go:122] "RemoveContainer" containerID="9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e" Nov 03 17:30:41.586436 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.586401 4539 scope.go:122] "RemoveContainer" containerID="8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd" Nov 03 17:30:41.586753 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:41.586732 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd\": container with ID starting with 8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd not found: ID does not exist" containerID="8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd" Nov 03 17:30:41.586860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.586765 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd"} err="failed to get container status \"8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd\": rpc error: code = NotFound desc = could not find container \"8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd\": container with ID starting with 8e5aa1a795cfe45563ea0304a81c3b3e69082d0da5f6a73012891d687d0d5edd not found: ID does not exist" Nov 03 17:30:41.586860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.586791 4539 scope.go:122] "RemoveContainer" containerID="9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e" Nov 03 17:30:41.587020 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:41.587002 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e\": container with ID starting with 9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e not found: ID does not exist" containerID="9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e" Nov 03 17:30:41.587089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.587039 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e"} err="failed to get container status \"9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e\": rpc error: code = NotFound desc = could not find container \"9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e\": container with ID starting with 9aded479cd55cde67147f9a9c68b8c1c14fea545534da972d1140a153010de8e not found: ID does not exist" Nov 03 17:30:41.629829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.629795 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb"] Nov 03 17:30:41.641871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.641837 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2492c4-openshift-default-784458cc77-r87mb"] Nov 03 17:30:41.781821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:41.781789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:42.479002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:42.478964 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80150068-7b02-4d59-84f7-1e23530dd995" path="/var/lib/kubelet/pods/80150068-7b02-4d59-84f7-1e23530dd995/volumes" Nov 03 17:30:42.941934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:42.941893 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:43.262737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:43.262639 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:43.426283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:43.426245 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:43.590474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:43.585008 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:43.738257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:43.738219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:44.182661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:44.182129 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:45.844082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:45.844046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:46.485583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:46.485544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:46.656089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:46.656053 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:47.283413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:47.283375 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:47.694535 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:47.694447 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:47.841403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:47.841350 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:48.584178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:48.584144 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:49.529203 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:49.529098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:49.880008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:49.879917 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:50.032242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:50.032202 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:50.411629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:50.411587 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:50.812830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:50.812798 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:51.270303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:51.270216 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:52.782083 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:52.782043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:54.182615 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:54.182571 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:54.330482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:54.330388 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:54.539078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:54.539026 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:55.420499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:55.420462 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:55.780077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:55.780041 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh"] Nov 03 17:30:55.780295 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:55.780258 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" podUID="dbf58250-f040-4089-9364-6600e22887ac" containerName="maas-api" containerID="cri-o://2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5" gracePeriod=30 Nov 03 17:30:55.910387 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:55.910347 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:56.280588 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:56.275393 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:30:56.350815 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.350787 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 17:30:56.415749 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.415714 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45\" (UniqueName: \"kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45\") pod \"dbf58250-f040-4089-9364-6600e22887ac\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " Nov 03 17:30:56.415972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.415787 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls\") pod \"dbf58250-f040-4089-9364-6600e22887ac\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " Nov 03 17:30:56.415972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.415848 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs\") pod \"dbf58250-f040-4089-9364-6600e22887ac\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " Nov 03 17:30:56.415972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.415919 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca\") pod \"dbf58250-f040-4089-9364-6600e22887ac\" (UID: \"dbf58250-f040-4089-9364-6600e22887ac\") " Nov 03 17:30:56.416714 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.416676 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca" pod "dbf58250-f040-4089-9364-6600e22887ac" (UID: "dbf58250-f040-4089-9364-6600e22887ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:56.419370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.419319 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls" pod "dbf58250-f040-4089-9364-6600e22887ac" (UID: "dbf58250-f040-4089-9364-6600e22887ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:56.419524 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.419396 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs" pod "dbf58250-f040-4089-9364-6600e22887ac" (UID: "dbf58250-f040-4089-9364-6600e22887ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:56.421024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.420996 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45" pod "dbf58250-f040-4089-9364-6600e22887ac" (UID: "dbf58250-f040-4089-9364-6600e22887ac"). InnerVolumeSpecName "kube-api-access-b8j45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:56.470626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.470593 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj"] Nov 03 17:30:56.470865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.470839 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" containerID="cri-o://25b1e721cb957d461ec873a994fec88dc4ab974d30b858e4634a12e218ea80bd" gracePeriod=30 Nov 03 17:30:56.521469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.518205 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8j45\" (UniqueName: \"kubernetes.io/projected/dbf58250-f040-4089-9364-6600e22887ac-kube-api-access-b8j45\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:56.521469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.518238 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:56.521469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.518252 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbf58250-f040-4089-9364-6600e22887ac-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:56.521469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.518264 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf58250-f040-4089-9364-6600e22887ac-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:30:56.570505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.570370 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:56.746220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.746184 4539 generic.go:356] "Generic (PLEG): container finished" podID="dbf58250-f040-4089-9364-6600e22887ac" containerID="2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5" exitCode=0 Nov 03 17:30:56.746491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.746265 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" Nov 03 17:30:56.746491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.746291 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" event={"ID":"dbf58250-f040-4089-9364-6600e22887ac","Type":"ContainerDied","Data":"2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5"} Nov 03 17:30:56.746491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.746393 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh" event={"ID":"dbf58250-f040-4089-9364-6600e22887ac","Type":"ContainerDied","Data":"b815b9cf103fc9092408b632dd4a305e83c241f2aefa44bbfc0f416e9a07f00a"} Nov 03 17:30:56.746491 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.746439 4539 scope.go:122] "RemoveContainer" containerID="2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5" Nov 03 17:30:56.757144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.757121 4539 scope.go:122] "RemoveContainer" containerID="2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5" Nov 03 17:30:56.757498 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:30:56.757477 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5\": container with ID starting with 2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5 not found: ID does not exist" containerID="2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5" Nov 03 17:30:56.757598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.757506 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5"} err="failed to get container status \"2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5\": rpc error: code = NotFound desc = could not find container \"2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5\": container with ID starting with 2553bc7b19a392e80596f4935b03d168002a753d434d52224d9e9d775d2e1ee5 not found: ID does not exist" Nov 03 17:30:56.801476 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.800302 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh"] Nov 03 17:30:56.811839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:56.811809 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-001802-5c8c6598-hmpqh"] Nov 03 17:30:57.065570 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:57.065540 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:57.751077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:57.751042 4539 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:30:57.751566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:57.751127 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:30:58.189052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:58.189017 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:30:58.469997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:58.469917 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf58250-f040-4089-9364-6600e22887ac" path="/var/lib/kubelet/pods/dbf58250-f040-4089-9364-6600e22887ac/volumes" Nov 03 17:30:59.887621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:30:59.887589 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:00.693251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:00.693211 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb"] Nov 03 17:31:00.693523 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:00.693446 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" containerID="cri-o://07ea29968aaab97c24d6bcc1c775012364d3440c6a85133c6c6aeccdc60b16c8" gracePeriod=30 Nov 03 17:31:01.154821 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.154774 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n"] Nov 03 17:31:01.155383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155367 4539 cpu_manager.go:414] "containerMap: removing container" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" Nov 03 17:31:01.155383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155380 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155396 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155401 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155408 4539 cpu_manager.go:414] "containerMap: removing container" podUID="dbf58250-f040-4089-9364-6600e22887ac" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155412 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf58250-f040-4089-9364-6600e22887ac" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155439 4539 cpu_manager.go:414] "containerMap: removing container" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155444 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155450 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerName="payload-pre-processing" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155456 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerName="payload-pre-processing" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155472 4539 cpu_manager.go:414] "containerMap: removing container" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerName="payload-processing" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155477 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerName="payload-processing" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155482 4539 cpu_manager.go:414] "containerMap: removing container" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155487 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155497 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155502 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155596 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b727cdc6-4114-46ef-be3a-fd5e4cd6a67a" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155604 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf58250-f040-4089-9364-6600e22887ac" containerName="maas-api" Nov 03 17:31:01.155597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155611 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.156550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155618 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f0db2d-138f-467c-9981-42b8a42ed55c" containerName="payload-pre-processing" Nov 03 17:31:01.156550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155626 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="589dfe3f-d37b-4c6e-8cfb-ed68c49c0e5d" containerName="payload-processing" Nov 03 17:31:01.156550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155632 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.156550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155639 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:01.156550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155646 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbf26c4-2f14-46ad-bd7e-c019d3e4d375" containerName="maas-api" Nov 03 17:31:01.156550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.155652 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" Nov 03 17:31:01.159358 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.159334 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.174729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.174702 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"openshift-service-ca.crt\"" Nov 03 17:31:01.174923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.174870 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"kube-root-ca.crt\"" Nov 03 17:31:01.182914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.182884 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n"] Nov 03 17:31:01.192473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.187008 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"rate-test-model-3b6f35-kserve-self-signed-certs\"" Nov 03 17:31:01.192473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.187062 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"default-dockercfg-hdcvc\"" Nov 03 17:31:01.250550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.250519 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:01.299301 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.299207 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.299559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.299368 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.299559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.299409 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.299559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.299547 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.299752 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.299734 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvms\" (UniqueName: \"kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.299817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.299772 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.401444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.401385 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvms\" (UniqueName: \"kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.401670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.401452 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.401670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.401504 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.401670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.401565 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.401670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.401590 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.401670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.401632 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.402040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.402020 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.402860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.402834 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.403069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.403042 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.404701 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.404675 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.406167 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.406109 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.431450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.428119 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvms\" (UniqueName: \"kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms\") pod \"rate-test-model-3b6f35-kserve-67cc487fc-z5f5n\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.480133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.480059 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:01.683699 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.683381 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:01.790480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.790417 4539 generic.go:356] "Generic (PLEG): container finished" podID="ea08502f-d143-4369-b374-ef63f987872e" containerID="25b1e721cb957d461ec873a994fec88dc4ab974d30b858e4634a12e218ea80bd" exitCode=0 Nov 03 17:31:01.790738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.790570 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" event={"ID":"ea08502f-d143-4369-b374-ef63f987872e","Type":"ContainerDied","Data":"25b1e721cb957d461ec873a994fec88dc4ab974d30b858e4634a12e218ea80bd"} Nov 03 17:31:01.830564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.830537 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 17:31:01.889984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.889952 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n"] Nov 03 17:31:01.939657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939582 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.939657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939650 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.939852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939723 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.939852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939765 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.939852 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939827 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.940033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939862 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.940033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939869 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.940033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.939899 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.940033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940000 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.940274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940053 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.940274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940179 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy\") pod \"ea08502f-d143-4369-b374-ef63f987872e\" (UID: \"ea08502f-d143-4369-b374-ef63f987872e\") " Nov 03 17:31:01.940397 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940277 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.940397 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940326 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.940608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940588 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:01.940781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940759 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.940884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940790 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.940884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940804 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.940884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940818 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea08502f-d143-4369-b374-ef63f987872e-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.940884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.940830 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.942751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.942725 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "kube-api-access-gt9rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:01.942842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.942808 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:01.943395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.943274 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:01.943652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:01.943625 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy" pod "ea08502f-d143-4369-b374-ef63f987872e" (UID: "ea08502f-d143-4369-b374-ef63f987872e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:02.042597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.042566 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea08502f-d143-4369-b374-ef63f987872e-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:02.042597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.042593 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9rm\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-kube-api-access-gt9rm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:02.042597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.042605 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea08502f-d143-4369-b374-ef63f987872e-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:02.042885 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.042614 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea08502f-d143-4369-b374-ef63f987872e-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:02.804660 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.804619 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" event={"ID":"ea08502f-d143-4369-b374-ef63f987872e","Type":"ContainerDied","Data":"20f1051bcbed2fcc5518c7b6188fda47ec6afd153e0763ef58417384d172ac6a"} Nov 03 17:31:02.805257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.804694 4539 scope.go:122] "RemoveContainer" containerID="25b1e721cb957d461ec873a994fec88dc4ab974d30b858e4634a12e218ea80bd" Nov 03 17:31:02.805257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.804866 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj" Nov 03 17:31:02.814445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.808971 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" event={"ID":"d1931cc8-edd5-401a-9589-61d65be8fdcf","Type":"ContainerStarted","Data":"ea7ddc7ca00f4ba44b17b2de120a21500c945a087b23b542068f613490629afd"} Nov 03 17:31:02.814445 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.809024 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" event={"ID":"d1931cc8-edd5-401a-9589-61d65be8fdcf","Type":"ContainerStarted","Data":"52bd7b730e41070f81d733099fdabaf2208a19c2f9f651b5d900a87bce4d2c36"} Nov 03 17:31:02.872492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.872449 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj"] Nov 03 17:31:02.883494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.883458 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-ffhsj"] Nov 03 17:31:02.909815 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.909778 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:02.916333 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:02.916277 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" podStartSLOduration=1.916261303 podStartE2EDuration="1.916261303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:02.913164191 +0000 UTC m=+4606.606435213" watchObservedRunningTime="2026-11-03 17:31:02.916261303 +0000 UTC m=+4606.609532325" Nov 03 17:31:03.067289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:03.067185 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:03.481514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:03.481410 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:03.536286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:03.536246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:03.697469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:03.697436 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:04.114133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:04.114093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:04.470330 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:04.470247 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea08502f-d143-4369-b374-ef63f987872e" path="/var/lib/kubelet/pods/ea08502f-d143-4369-b374-ef63f987872e/volumes" Nov 03 17:31:04.599531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:04.599496 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:05.448412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:05.448378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:05.838546 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:05.838515 4539 generic.go:356] "Generic (PLEG): container finished" podID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerID="07ea29968aaab97c24d6bcc1c775012364d3440c6a85133c6c6aeccdc60b16c8" exitCode=0 Nov 03 17:31:05.838742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:05.838592 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" event={"ID":"e6ef7b37-de64-408e-b169-8c68842e9cb2","Type":"ContainerDied","Data":"07ea29968aaab97c24d6bcc1c775012364d3440c6a85133c6c6aeccdc60b16c8"} Nov 03 17:31:05.986590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:05.986520 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 17:31:06.104773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.104731 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.104773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.104774 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.104807 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.104829 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.104999 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105090 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105179 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105089 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:06.105341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105214 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:06.105341 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105227 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105475 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs\") pod \"e6ef7b37-de64-408e-b169-8c68842e9cb2\" (UID: \"e6ef7b37-de64-408e-b169-8c68842e9cb2\") " Nov 03 17:31:06.105614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105546 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:06.105675 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.105608 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:06.106060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.106028 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:06.106308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.106287 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ef7b37-de64-408e-b169-8c68842e9cb2-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.106380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.106314 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.106380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.106329 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.106380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.106342 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.106380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.106372 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.107680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.107650 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:06.107803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.107727 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:06.107998 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.107976 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:06.108133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.108056 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt" pod "e6ef7b37-de64-408e-b169-8c68842e9cb2" (UID: "e6ef7b37-de64-408e-b169-8c68842e9cb2"). InnerVolumeSpecName "kube-api-access-wmxzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:06.207173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.207133 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.207173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.207167 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.207173 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.207177 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxzt\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-kube-api-access-wmxzt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.207478 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.207186 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ef7b37-de64-408e-b169-8c68842e9cb2-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:06.847363 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.847336 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" Nov 03 17:31:06.847832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.847335 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb" event={"ID":"e6ef7b37-de64-408e-b169-8c68842e9cb2","Type":"ContainerDied","Data":"3482146cc841e879ee2d5a726ccba467e8923f380a91e17b3f8911490c1d2ae4"} Nov 03 17:31:06.847832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.847511 4539 scope.go:122] "RemoveContainer" containerID="07ea29968aaab97c24d6bcc1c775012364d3440c6a85133c6c6aeccdc60b16c8" Nov 03 17:31:06.939673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.939642 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb"] Nov 03 17:31:06.951816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:06.951782 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-f8wsb"] Nov 03 17:31:07.265698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:07.265611 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:08.470486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:08.470448 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" path="/var/lib/kubelet/pods/e6ef7b37-de64-408e-b169-8c68842e9cb2/volumes" Nov 03 17:31:08.946276 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:08.946248 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:09.515761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:09.515714 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm"] Nov 03 17:31:09.528274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:09.528241 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp"] Nov 03 17:31:09.540813 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:09.540779 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c"] Nov 03 17:31:09.672538 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:09.672506 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:10.561071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:10.561035 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:11.480917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:11.480879 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:11.494720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:11.494685 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:11.900119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:11.900086 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:31:11.900997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:11.900969 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:12.528261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:12.528183 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:13.041419 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:13.041383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:13.185403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:13.185370 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:13.746336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:13.746301 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:14.185080 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:14.185043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:14.335163 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:14.335121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:15.468195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:15.468159 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:16.118986 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:16.118944 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:17.295254 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.295212 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:17.437360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.437320 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v"] Nov 03 17:31:17.437711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.437639 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" podUID="266557c9-7986-414b-8dbe-d373417c4b12" containerName="maas-api" containerID="cri-o://be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441" gracePeriod=30 Nov 03 17:31:17.532977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.532935 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:17.895459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.894970 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:31:17.952470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.949641 4539 generic.go:356] "Generic (PLEG): container finished" podID="266557c9-7986-414b-8dbe-d373417c4b12" containerID="be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441" exitCode=0 Nov 03 17:31:17.952470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.949707 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" event={"ID":"266557c9-7986-414b-8dbe-d373417c4b12","Type":"ContainerDied","Data":"be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441"} Nov 03 17:31:17.952470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.949759 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" event={"ID":"266557c9-7986-414b-8dbe-d373417c4b12","Type":"ContainerDied","Data":"0a9593ee0a2f34a233b68b242d844ee8ba69f2abc0d825274a8ff09572816180"} Nov 03 17:31:17.952470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.949788 4539 scope.go:122] "RemoveContainer" containerID="be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441" Nov 03 17:31:17.952470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.949953 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v" Nov 03 17:31:17.964578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.964553 4539 scope.go:122] "RemoveContainer" containerID="be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441" Nov 03 17:31:17.964979 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:17.964951 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441\": container with ID starting with be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441 not found: ID does not exist" containerID="be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441" Nov 03 17:31:17.965052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.964995 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441"} err="failed to get container status \"be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441\": rpc error: code = NotFound desc = could not find container \"be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441\": container with ID starting with be125e61836cd30800be649c8431feab474505c35940d3bedf756aff17423441 not found: ID does not exist" Nov 03 17:31:17.968534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.968504 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4\" (UniqueName: \"kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4\") pod \"266557c9-7986-414b-8dbe-d373417c4b12\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " Nov 03 17:31:17.968667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.968537 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls\") pod \"266557c9-7986-414b-8dbe-d373417c4b12\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " Nov 03 17:31:17.968667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.968593 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs\") pod \"266557c9-7986-414b-8dbe-d373417c4b12\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " Nov 03 17:31:17.968798 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.968693 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca\") pod \"266557c9-7986-414b-8dbe-d373417c4b12\" (UID: \"266557c9-7986-414b-8dbe-d373417c4b12\") " Nov 03 17:31:17.970011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.969966 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca" pod "266557c9-7986-414b-8dbe-d373417c4b12" (UID: "266557c9-7986-414b-8dbe-d373417c4b12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:17.973127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.972978 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs" pod "266557c9-7986-414b-8dbe-d373417c4b12" (UID: "266557c9-7986-414b-8dbe-d373417c4b12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:17.973456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.973409 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls" pod "266557c9-7986-414b-8dbe-d373417c4b12" (UID: "266557c9-7986-414b-8dbe-d373417c4b12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:17.973938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:17.973923 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4" pod "266557c9-7986-414b-8dbe-d373417c4b12" (UID: "266557c9-7986-414b-8dbe-d373417c4b12"). InnerVolumeSpecName "kube-api-access-5wqn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:18.070382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.070344 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266557c9-7986-414b-8dbe-d373417c4b12-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:18.070382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.070380 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wqn4\" (UniqueName: \"kubernetes.io/projected/266557c9-7986-414b-8dbe-d373417c4b12-kube-api-access-5wqn4\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:18.070382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.070391 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:18.070692 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.070400 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266557c9-7986-414b-8dbe-d373417c4b12-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:18.187112 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.182378 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:18.297142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.297111 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v"] Nov 03 17:31:18.301652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.301617 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6046c2-77cf7dcc4d-srj6v"] Nov 03 17:31:18.469837 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:18.469755 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266557c9-7986-414b-8dbe-d373417c4b12" path="/var/lib/kubelet/pods/266557c9-7986-414b-8dbe-d373417c4b12/volumes" Nov 03 17:31:19.673725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:19.673691 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:21.399546 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:21.399465 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:21.692685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:21.692573 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:21.865877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:21.865842 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:23.158462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:23.156315 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:23.683922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:23.683889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:24.540552 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.540486 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" containerID="cri-o://ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc" gracePeriod=45 Nov 03 17:31:24.556076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.555991 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" containerID="cri-o://22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402" gracePeriod=45 Nov 03 17:31:24.570853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.570820 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" containerID="cri-o://867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e" gracePeriod=45 Nov 03 17:31:24.939748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.939712 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 17:31:24.945128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.945072 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 17:31:24.949133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.949108 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 17:31:24.977294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.977208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:24.992028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.991989 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn\" (UniqueName: \"kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn\") pod \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " Nov 03 17:31:24.992332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.992312 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs\") pod \"5fe3689f-246d-4bb3-b180-73038521aadd\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " Nov 03 17:31:25.057612 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057575 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache\") pod \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " Nov 03 17:31:25.057822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057639 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir\") pod \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " Nov 03 17:31:25.057822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057697 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache\") pod \"5fe3689f-246d-4bb3-b180-73038521aadd\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " Nov 03 17:31:25.057822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057770 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm\") pod \"5fe3689f-246d-4bb3-b180-73038521aadd\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " Nov 03 17:31:25.057822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057812 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir\") pod \"5fe3689f-246d-4bb3-b180-73038521aadd\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " Nov 03 17:31:25.058064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057877 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs\") pod \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " Nov 03 17:31:25.058064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057914 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home\") pod \"5fe3689f-246d-4bb3-b180-73038521aadd\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " Nov 03 17:31:25.058064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.057992 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home\") pod \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " Nov 03 17:31:25.058213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058062 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm\") pod \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " Nov 03 17:31:25.058213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058148 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir\") pod \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " Nov 03 17:31:25.058213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058195 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs\") pod \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " Nov 03 17:31:25.058378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.035648 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" Nov 03 17:31:25.058378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058262 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x\" (UniqueName: \"kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x\") pod \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " Nov 03 17:31:25.058378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058305 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4\") pod \"5fe3689f-246d-4bb3-b180-73038521aadd\" (UID: \"5fe3689f-246d-4bb3-b180-73038521aadd\") " Nov 03 17:31:25.058378 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058371 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm\") pod \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " Nov 03 17:31:25.058621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058453 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache\") pod \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\" (UID: \"cb525d2d-1f9c-4354-90be-18439f5ad7a4\") " Nov 03 17:31:25.058621 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058498 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home\") pod \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\" (UID: \"ef6de8c8-3034-4a43-8e58-04dab26ee9de\") " Nov 03 17:31:25.058886 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.035442 4539 generic.go:356] "Generic (PLEG): container finished" podID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerID="22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402" exitCode=0 Nov 03 17:31:25.062969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.035488 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" event={"ID":"ef6de8c8-3034-4a43-8e58-04dab26ee9de","Type":"ContainerDied","Data":"22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402"} Nov 03 17:31:25.062969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.062449 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp" event={"ID":"ef6de8c8-3034-4a43-8e58-04dab26ee9de","Type":"ContainerDied","Data":"6ae935189a183808350d4bf929ac08e1d41ce2da798e956dbeb2947b78ab641f"} Nov 03 17:31:25.062969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.062491 4539 scope.go:122] "RemoveContainer" containerID="22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402" Nov 03 17:31:25.064015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:24.995400 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs" pod "5fe3689f-246d-4bb3-b180-73038521aadd" (UID: "5fe3689f-246d-4bb3-b180-73038521aadd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:25.064188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.037059 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn" pod "cb525d2d-1f9c-4354-90be-18439f5ad7a4" (UID: "cb525d2d-1f9c-4354-90be-18439f5ad7a4"). InnerVolumeSpecName "kube-api-access-dj7zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:25.064310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058213 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir" pod "5fe3689f-246d-4bb3-b180-73038521aadd" (UID: "5fe3689f-246d-4bb3-b180-73038521aadd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.064415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.058858 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache" pod "5fe3689f-246d-4bb3-b180-73038521aadd" (UID: "5fe3689f-246d-4bb3-b180-73038521aadd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.064554 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.059666 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home" pod "ef6de8c8-3034-4a43-8e58-04dab26ee9de" (UID: "ef6de8c8-3034-4a43-8e58-04dab26ee9de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.064649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.061561 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm" pod "5fe3689f-246d-4bb3-b180-73038521aadd" (UID: "5fe3689f-246d-4bb3-b180-73038521aadd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.064735 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.061942 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs" pod "ef6de8c8-3034-4a43-8e58-04dab26ee9de" (UID: "ef6de8c8-3034-4a43-8e58-04dab26ee9de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:25.064856 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.064822 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir" pod "cb525d2d-1f9c-4354-90be-18439f5ad7a4" (UID: "cb525d2d-1f9c-4354-90be-18439f5ad7a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.064957 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.062343 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir" pod "ef6de8c8-3034-4a43-8e58-04dab26ee9de" (UID: "ef6de8c8-3034-4a43-8e58-04dab26ee9de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.065044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.063634 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home" pod "cb525d2d-1f9c-4354-90be-18439f5ad7a4" (UID: "cb525d2d-1f9c-4354-90be-18439f5ad7a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.065128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.063651 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache" pod "cb525d2d-1f9c-4354-90be-18439f5ad7a4" (UID: "cb525d2d-1f9c-4354-90be-18439f5ad7a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.065209 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.063840 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home" pod "5fe3689f-246d-4bb3-b180-73038521aadd" (UID: "5fe3689f-246d-4bb3-b180-73038521aadd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.065301 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.065267 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm" pod "ef6de8c8-3034-4a43-8e58-04dab26ee9de" (UID: "ef6de8c8-3034-4a43-8e58-04dab26ee9de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.065373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.064597 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x" pod "ef6de8c8-3034-4a43-8e58-04dab26ee9de" (UID: "ef6de8c8-3034-4a43-8e58-04dab26ee9de"). InnerVolumeSpecName "kube-api-access-jp85x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:25.067121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.067069 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs" pod "cb525d2d-1f9c-4354-90be-18439f5ad7a4" (UID: "cb525d2d-1f9c-4354-90be-18439f5ad7a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:25.067121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.067095 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4" pod "5fe3689f-246d-4bb3-b180-73038521aadd" (UID: "5fe3689f-246d-4bb3-b180-73038521aadd"). InnerVolumeSpecName "kube-api-access-dq7b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:25.068305 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.068280 4539 generic.go:356] "Generic (PLEG): container finished" podID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerID="867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e" exitCode=0 Nov 03 17:31:25.068730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.068410 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" event={"ID":"cb525d2d-1f9c-4354-90be-18439f5ad7a4","Type":"ContainerDied","Data":"867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e"} Nov 03 17:31:25.068730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.068493 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" event={"ID":"cb525d2d-1f9c-4354-90be-18439f5ad7a4","Type":"ContainerDied","Data":"ac5e7daff2590d88e7c431e6b5d6e37c6c6d1c875386d42503dfcbbe36e9c83e"} Nov 03 17:31:25.068730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.068580 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c" Nov 03 17:31:25.070920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.070894 4539 generic.go:356] "Generic (PLEG): container finished" podID="5fe3689f-246d-4bb3-b180-73038521aadd" containerID="ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc" exitCode=0 Nov 03 17:31:25.071143 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.071122 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" event={"ID":"5fe3689f-246d-4bb3-b180-73038521aadd","Type":"ContainerDied","Data":"ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc"} Nov 03 17:31:25.071281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.071265 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" event={"ID":"5fe3689f-246d-4bb3-b180-73038521aadd","Type":"ContainerDied","Data":"d28ce3607aa31a826265113b6e73fc2a3a5b115630d639e06e8a52960a1c6861"} Nov 03 17:31:25.071359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.071301 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm" Nov 03 17:31:25.072989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.072964 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm" pod "cb525d2d-1f9c-4354-90be-18439f5ad7a4" (UID: "cb525d2d-1f9c-4354-90be-18439f5ad7a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.073207 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.073185 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache" pod "ef6de8c8-3034-4a43-8e58-04dab26ee9de" (UID: "ef6de8c8-3034-4a43-8e58-04dab26ee9de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:25.078342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.078322 4539 scope.go:122] "RemoveContainer" containerID="22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402" Nov 03 17:31:25.078803 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:25.078715 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402\": container with ID starting with 22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402 not found: ID does not exist" containerID="22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402" Nov 03 17:31:25.078803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.078750 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402"} err="failed to get container status \"22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402\": rpc error: code = NotFound desc = could not find container \"22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402\": container with ID starting with 22fe967231d38eef7b3366c0fbc094bd8ad69a7698e03e4cd035eddb9e3f4402 not found: ID does not exist" Nov 03 17:31:25.078803 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.078775 4539 scope.go:122] "RemoveContainer" containerID="867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e" Nov 03 17:31:25.094932 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.094906 4539 scope.go:122] "RemoveContainer" containerID="867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e" Nov 03 17:31:25.095291 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:25.095264 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e\": container with ID starting with 867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e not found: ID does not exist" containerID="867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e" Nov 03 17:31:25.095399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.095339 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e"} err="failed to get container status \"867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e\": rpc error: code = NotFound desc = could not find container \"867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e\": container with ID starting with 867da831f772c0f2769ba94143728b9a665b32f06370cc789cab6727d5c7f90e not found: ID does not exist" Nov 03 17:31:25.095399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.095366 4539 scope.go:122] "RemoveContainer" containerID="ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc" Nov 03 17:31:25.112968 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.112935 4539 scope.go:122] "RemoveContainer" containerID="ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc" Nov 03 17:31:25.113637 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:25.113609 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc\": container with ID starting with ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc not found: ID does not exist" containerID="ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc" Nov 03 17:31:25.113768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.113648 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc"} err="failed to get container status \"ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc\": rpc error: code = NotFound desc = could not find container \"ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc\": container with ID starting with ad3f369d42a548b0ac48da3d59a6b62f98b31c26ae4587bee8d2ae1c6b04bacc not found: ID does not exist" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160547 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160578 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160588 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160596 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160604 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160610 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160617 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160624 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb525d2d-1f9c-4354-90be-18439f5ad7a4-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160634 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp85x\" (UniqueName: \"kubernetes.io/projected/ef6de8c8-3034-4a43-8e58-04dab26ee9de-kube-api-access-jp85x\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.160633 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160643 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/5fe3689f-246d-4bb3-b180-73038521aadd-kube-api-access-dq7b4\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160654 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160662 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb525d2d-1f9c-4354-90be-18439f5ad7a4-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160669 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160677 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7zn\" (UniqueName: \"kubernetes.io/projected/cb525d2d-1f9c-4354-90be-18439f5ad7a4-kube-api-access-dj7zn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160684 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3689f-246d-4bb3-b180-73038521aadd-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160692 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160699 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6de8c8-3034-4a43-8e58-04dab26ee9de-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.161275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.160705 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3689f-246d-4bb3-b180-73038521aadd-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.172488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.172458 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm"] Nov 03 17:31:25.178758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.178731 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-2-w2-7519b7-kserve-7df99fb68f-nvhtm"] Nov 03 17:31:25.405088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.405049 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp"] Nov 03 17:31:25.410332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.410301 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/premium-w2-7519b7-kserve-55bdb789d7-mqjsp"] Nov 03 17:31:25.445061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.444898 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c"] Nov 03 17:31:25.449029 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.449007 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/simulator-w2-7519b7-kserve-6b94bb8b78-d7s2c"] Nov 03 17:31:25.619483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.619450 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb"] Nov 03 17:31:25.620323 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620297 4539 cpu_manager.go:414] "containerMap: removing container" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" Nov 03 17:31:25.620323 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620323 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620339 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620346 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620361 4539 cpu_manager.go:414] "containerMap: removing container" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620370 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620392 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620399 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620411 4539 cpu_manager.go:414] "containerMap: removing container" podUID="266557c9-7986-414b-8dbe-d373417c4b12" containerName="maas-api" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620418 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266557c9-7986-414b-8dbe-d373417c4b12" containerName="maas-api" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620463 4539 cpu_manager.go:414] "containerMap: removing container" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620470 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb776568-4552-40bd-9238-288382562539" containerName="cleanup" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620496 4539 cpu_manager.go:414] "containerMap: removing container" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620503 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620522 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" Nov 03 17:31:25.620550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620529 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620683 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" containerName="main" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620701 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ef7b37-de64-408e-b169-8c68842e9cb2" containerName="istio-proxy" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620713 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea08502f-d143-4369-b374-ef63f987872e" containerName="istio-proxy" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620727 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" containerName="main" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620737 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="266557c9-7986-414b-8dbe-d373417c4b12" containerName="maas-api" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620752 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="80150068-7b02-4d59-84f7-1e23530dd995" containerName="istio-proxy" Nov 03 17:31:25.621308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.620760 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" containerName="main" Nov 03 17:31:25.626651 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.626624 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.642626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.642600 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"kube-root-ca.crt\"" Nov 03 17:31:25.642867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.642618 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"openshift-service-ca.crt\"" Nov 03 17:31:25.642867 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.642640 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"rate-test-model-5a5ab1-kserve-self-signed-certs\"" Nov 03 17:31:25.642989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.642711 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"default-dockercfg-cwbfj\"" Nov 03 17:31:25.647461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.647416 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb"] Nov 03 17:31:25.666900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.666859 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bh6k\" (UniqueName: \"kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.667115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.667092 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.667293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.667269 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.667418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.667400 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.667514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.667493 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.667557 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.667537 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.714863 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.714715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:25.768376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768329 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768414 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768452 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768473 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768497 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bh6k\" (UniqueName: \"kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768589 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768828 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.768923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.768888 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.769526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.769499 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.771481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.771458 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.772359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.772337 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.796395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.796363 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bh6k\" (UniqueName: \"kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k\") pod \"rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:25.945654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:25.945618 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:26.158387 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:26.158350 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:31:26.263942 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:26.263908 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb"] Nov 03 17:31:26.264753 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:31:26.264728 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac86edc8_1098_4b62_a332_0a9d7865c52f.slice/crio-3d59caecabbe283f13d2853097ae737542ea7ae92377f0b3b6e307c32e1b5632 WatchSource:0}: Error finding container 3d59caecabbe283f13d2853097ae737542ea7ae92377f0b3b6e307c32e1b5632: Status 404 returned error can't find the container with id 3d59caecabbe283f13d2853097ae737542ea7ae92377f0b3b6e307c32e1b5632 Nov 03 17:31:26.469164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:26.469126 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe3689f-246d-4bb3-b180-73038521aadd" path="/var/lib/kubelet/pods/5fe3689f-246d-4bb3-b180-73038521aadd/volumes" Nov 03 17:31:26.469678 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:26.469653 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb525d2d-1f9c-4354-90be-18439f5ad7a4" path="/var/lib/kubelet/pods/cb525d2d-1f9c-4354-90be-18439f5ad7a4/volumes" Nov 03 17:31:26.470074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:26.470061 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6de8c8-3034-4a43-8e58-04dab26ee9de" path="/var/lib/kubelet/pods/ef6de8c8-3034-4a43-8e58-04dab26ee9de/volumes" Nov 03 17:31:26.560230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:26.560194 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:27.091804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:27.091769 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" event={"ID":"ac86edc8-1098-4b62-a332-0a9d7865c52f","Type":"ContainerStarted","Data":"114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00"} Nov 03 17:31:27.091804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:27.091815 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" event={"ID":"ac86edc8-1098-4b62-a332-0a9d7865c52f","Type":"ContainerStarted","Data":"3d59caecabbe283f13d2853097ae737542ea7ae92377f0b3b6e307c32e1b5632"} Nov 03 17:31:27.136345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:27.136272 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" podStartSLOduration=2.136250169 podStartE2EDuration="2.136250169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:27.133594168 +0000 UTC m=+4630.826865207" watchObservedRunningTime="2026-11-03 17:31:27.136250169 +0000 UTC m=+4630.829521193" Nov 03 17:31:27.946082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:27.946037 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:28.226844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:28.226735 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:28.649925 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:28.649722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:28.847180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:28.847134 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:29.327415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:29.327365 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:29.658447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:29.658311 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl"] Nov 03 17:31:29.658688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:29.658659 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" containerID="cri-o://b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1" gracePeriod=30 Nov 03 17:31:29.841989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:29.841949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:30.003342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:30.003239 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:30.557743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:30.557711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:30.693618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:30.693580 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:31.683918 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:31.683875 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:33.269377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:33.269329 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:34.813635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.813593 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:34.938493 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.938388 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:31:34.991506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991472 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991519 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991572 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991594 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991626 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991648 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991758 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991777 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.991913 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.991824 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy\") pod \"c641e1c4-3301-4c83-85a1-f17596684ea7\" (UID: \"c641e1c4-3301-4c83-85a1-f17596684ea7\") " Nov 03 17:31:34.992808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.992780 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:34.993145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.993125 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:34.993546 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.993520 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:34.993715 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.993596 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:34.993860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.993779 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:34.999937 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.999894 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:34.999937 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.999911 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:35.000131 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:34.999941 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "kube-api-access-wpn62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:35.000203 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.000174 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token" pod "c641e1c4-3301-4c83-85a1-f17596684ea7" (UID: "c641e1c4-3301-4c83-85a1-f17596684ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:35.093318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093279 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093314 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c641e1c4-3301-4c83-85a1-f17596684ea7-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093323 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093330 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093340 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpn62\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-kube-api-access-wpn62\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093350 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093359 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093367 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.093626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.093375 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c641e1c4-3301-4c83-85a1-f17596684ea7-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:35.162901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.162855 4539 generic.go:356] "Generic (PLEG): container finished" podID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerID="b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1" exitCode=0 Nov 03 17:31:35.163127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.162904 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" Nov 03 17:31:35.163127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.162943 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" event={"ID":"c641e1c4-3301-4c83-85a1-f17596684ea7","Type":"ContainerDied","Data":"b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1"} Nov 03 17:31:35.163127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.163005 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl" event={"ID":"c641e1c4-3301-4c83-85a1-f17596684ea7","Type":"ContainerDied","Data":"66b91831f8803948aa69b865fada171ac98f3b79e31f8c43461f34f1ed756e49"} Nov 03 17:31:35.163127 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.163028 4539 scope.go:122] "RemoveContainer" containerID="b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1" Nov 03 17:31:35.175976 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.175947 4539 scope.go:122] "RemoveContainer" containerID="3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178" Nov 03 17:31:35.187470 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.187406 4539 scope.go:122] "RemoveContainer" containerID="b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1" Nov 03 17:31:35.187845 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:35.187811 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1\": container with ID starting with b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1 not found: ID does not exist" containerID="b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1" Nov 03 17:31:35.187964 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.187847 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1"} err="failed to get container status \"b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1\": rpc error: code = NotFound desc = could not find container \"b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1\": container with ID starting with b24cb8eac2f454ec8716ead925805b4e0dd6ec42d9d7775d666999257b8219d1 not found: ID does not exist" Nov 03 17:31:35.187964 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.187868 4539 scope.go:122] "RemoveContainer" containerID="3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178" Nov 03 17:31:35.188144 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:35.188125 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178\": container with ID starting with 3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178 not found: ID does not exist" containerID="3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178" Nov 03 17:31:35.188221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.188147 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178"} err="failed to get container status \"3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178\": rpc error: code = NotFound desc = could not find container \"3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178\": container with ID starting with 3eec2c3988b3c06a79d764938a6f757991594dc3721ea171d4bb020e64225178 not found: ID does not exist" Nov 03 17:31:35.216272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.216179 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl"] Nov 03 17:31:35.219583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.219553 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-wh8pl"] Nov 03 17:31:35.946410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.946366 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:35.960044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:35.960014 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:36.184233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:36.184201 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:31:36.479600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:36.479565 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:36.484033 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:36.484001 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" path="/var/lib/kubelet/pods/c641e1c4-3301-4c83-85a1-f17596684ea7/volumes" Nov 03 17:31:36.668641 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:36.668607 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:37.844016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:37.843984 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:38.241465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:38.241359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:38.498739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:38.498659 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:39.792701 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:39.792663 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:41.292659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.292623 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:41.528304 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.528267 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg"] Nov 03 17:31:41.529051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.529024 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" Nov 03 17:31:41.529051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.529046 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" Nov 03 17:31:41.529051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.529060 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" Nov 03 17:31:41.529332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.529066 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" Nov 03 17:31:41.529332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.529177 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" Nov 03 17:31:41.534117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.534095 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.552703 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.552678 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"kube-root-ca.crt\"" Nov 03 17:31:41.564624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.564599 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"openshift-service-ca.crt\"" Nov 03 17:31:41.565034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.565011 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"default-dockercfg-66t2h\"" Nov 03 17:31:41.565164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.565096 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c6f303\"/\"rate-test-model-c6f303-kserve-self-signed-certs\"" Nov 03 17:31:41.568980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.568959 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg"] Nov 03 17:31:41.634840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.634807 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:41.675233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.675197 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.675494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.675248 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqgn\" (UniqueName: \"kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.675494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.675273 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.675494 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.675415 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.675659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.675634 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.675759 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.675660 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.780449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780202 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.780449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780256 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.780449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780326 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.780449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780356 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqgn\" (UniqueName: \"kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.780449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780378 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.780877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780464 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.784450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.780987 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.784450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.781643 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.784450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.781890 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.787270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.787241 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.788004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.787981 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.842704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.842623 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqgn\" (UniqueName: \"kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn\") pod \"rate-test-model-c6f303-kserve-86d777466f-vdwrg\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:41.859566 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:41.859530 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:42.205912 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:42.205817 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg"] Nov 03 17:31:42.223493 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:31:42.223461 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54fe1cf4_c205_43e8_b58f_8df0023999c0.slice/crio-91090e3f5c6248976bdb83b8420697891fe67e33f7b1f2bdd3d03b7942e14b9a WatchSource:0}: Error finding container 91090e3f5c6248976bdb83b8420697891fe67e33f7b1f2bdd3d03b7942e14b9a: Status 404 returned error can't find the container with id 91090e3f5c6248976bdb83b8420697891fe67e33f7b1f2bdd3d03b7942e14b9a Nov 03 17:31:42.871575 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:42.871504 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:43.232283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:43.232181 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" event={"ID":"54fe1cf4-c205-43e8-b58f-8df0023999c0","Type":"ContainerStarted","Data":"645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b"} Nov 03 17:31:43.232580 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:43.232555 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" event={"ID":"54fe1cf4-c205-43e8-b58f-8df0023999c0","Type":"ContainerStarted","Data":"91090e3f5c6248976bdb83b8420697891fe67e33f7b1f2bdd3d03b7942e14b9a"} Nov 03 17:31:43.300496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:43.300403 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" podStartSLOduration=2.300380196 podStartE2EDuration="2.300380196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:43.297750114 +0000 UTC m=+4646.991021137" watchObservedRunningTime="2026-11-03 17:31:43.300380196 +0000 UTC m=+4646.993651219" Nov 03 17:31:43.494375 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:43.492099 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:43.860180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:43.860142 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:44.496878 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:44.496843 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:44.652594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:44.652557 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:45.081717 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:45.081676 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:45.253861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:45.250540 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:45.834969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:45.834909 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:46.494166 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:46.494127 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:48.097440 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:48.097327 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:48.548072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:48.548039 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:49.271449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:49.270938 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:49.724643 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:49.724610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:50.955178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:50.955143 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:51.860182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:51.860147 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:51.878244 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:51.878218 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:52.332530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:52.332502 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:31:52.651020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:52.650890 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:53.492307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:53.492273 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:53.950537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:53.950503 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:54.386273 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:54.386232 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:54.583041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:54.583000 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:54.788881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:54.788841 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:55.745274 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:55.745244 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:55.885670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:55.885634 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:55.937618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:55.937555 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t"] Nov 03 17:31:55.937883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:55.937816 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" podUID="a139fc2b-2df8-44dd-9356-2cefc49d160d" containerName="maas-api" containerID="cri-o://68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a" gracePeriod=30 Nov 03 17:31:56.041851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.041804 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:56.269331 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:56.269290 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:31:56.365460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.363178 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 17:31:56.371459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.367223 4539 generic.go:356] "Generic (PLEG): container finished" podID="a139fc2b-2df8-44dd-9356-2cefc49d160d" containerID="68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a" exitCode=0 Nov 03 17:31:56.371459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.367476 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" event={"ID":"a139fc2b-2df8-44dd-9356-2cefc49d160d","Type":"ContainerDied","Data":"68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a"} Nov 03 17:31:56.371459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.367534 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" event={"ID":"a139fc2b-2df8-44dd-9356-2cefc49d160d","Type":"ContainerDied","Data":"201b13625b94b03e2d60fbdabad81b1d409001c8aec3d8a2d3904c8f9dbd3bc4"} Nov 03 17:31:56.371459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.367566 4539 scope.go:122] "RemoveContainer" containerID="68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a" Nov 03 17:31:56.371459 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.367721 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t" Nov 03 17:31:56.388165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.388141 4539 scope.go:122] "RemoveContainer" containerID="68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a" Nov 03 17:31:56.389192 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:31:56.389162 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a\": container with ID starting with 68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a not found: ID does not exist" containerID="68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a" Nov 03 17:31:56.389358 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.389334 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a"} err="failed to get container status \"68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a\": rpc error: code = NotFound desc = could not find container \"68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a\": container with ID starting with 68352005bc7c833fe749bb30c6d1923474ee5ae64bbe37c773398352be0dd76a not found: ID does not exist" Nov 03 17:31:56.430396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.430359 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm\" (UniqueName: \"kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm\") pod \"a139fc2b-2df8-44dd-9356-2cefc49d160d\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " Nov 03 17:31:56.430641 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.430466 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls\") pod \"a139fc2b-2df8-44dd-9356-2cefc49d160d\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " Nov 03 17:31:56.430724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.430637 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca\") pod \"a139fc2b-2df8-44dd-9356-2cefc49d160d\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " Nov 03 17:31:56.430785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.430726 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs\") pod \"a139fc2b-2df8-44dd-9356-2cefc49d160d\" (UID: \"a139fc2b-2df8-44dd-9356-2cefc49d160d\") " Nov 03 17:31:56.432859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.432823 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca" pod "a139fc2b-2df8-44dd-9356-2cefc49d160d" (UID: "a139fc2b-2df8-44dd-9356-2cefc49d160d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:56.437572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.437538 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs" pod "a139fc2b-2df8-44dd-9356-2cefc49d160d" (UID: "a139fc2b-2df8-44dd-9356-2cefc49d160d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:56.447393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.447358 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls" pod "a139fc2b-2df8-44dd-9356-2cefc49d160d" (UID: "a139fc2b-2df8-44dd-9356-2cefc49d160d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:56.447555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.447358 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm" pod "a139fc2b-2df8-44dd-9356-2cefc49d160d" (UID: "a139fc2b-2df8-44dd-9356-2cefc49d160d"). InnerVolumeSpecName "kube-api-access-pwhrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:56.532262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.532194 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:56.532262 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.532261 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhrm\" (UniqueName: \"kubernetes.io/projected/a139fc2b-2df8-44dd-9356-2cefc49d160d-kube-api-access-pwhrm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:56.532609 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.532278 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a139fc2b-2df8-44dd-9356-2cefc49d160d-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:56.532609 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.532293 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a139fc2b-2df8-44dd-9356-2cefc49d160d-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:31:56.674628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.674547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:56.717578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.717541 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t"] Nov 03 17:31:56.722758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.722729 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2b49df-578d8f8dd5-g569t"] Nov 03 17:31:56.851686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:56.851636 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:57.006667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:57.006563 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:57.166005 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:57.165977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:31:58.469389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:58.469344 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a139fc2b-2df8-44dd-9356-2cefc49d160d" path="/var/lib/kubelet/pods/a139fc2b-2df8-44dd-9356-2cefc49d160d/volumes" Nov 03 17:31:58.595806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:31:58.595768 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:00.183267 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:00.175687 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:00.328882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:00.328811 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:00.480911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:00.480796 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:01.021607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:01.021571 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:01.404101 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:01.404060 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:01.834381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:01.834349 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:01.978923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:01.978892 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:03.495312 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:03.495279 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:05.179857 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:05.179794 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:06.829121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:06.829074 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:07.084171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:07.084090 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:08.366258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:08.366202 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:09.838569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:09.838533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:11.345018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:11.344980 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:12.869168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:12.869078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:14.545361 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:14.545324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:15.806988 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:15.806948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:16.225238 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:16.225121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:17.417183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:17.417133 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:17.977306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:17.977265 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:19.487152 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:19.487117 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:21.194526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:21.194487 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:22.134914 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:22.134864 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:22.279657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:22.279624 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:22.510629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:22.510538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:23.180783 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:23.180750 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:24.685213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:24.685175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:26.157534 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:32:26.157489 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:32:26.417379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:26.416694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:27.963724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:27.963650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:29.378933 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:29.378897 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:29.528110 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:29.528075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:31.167806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:31.167770 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:32.498593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:32.498555 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:33.165920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:33.165881 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:34.733702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:34.733662 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:35.158213 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:35.158173 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:35.414164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:35.414069 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:36.494472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:36.494435 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:37.988270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:37.988233 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:39.471905 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:39.471878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:41.137649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:41.137609 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:42.700197 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:42.700169 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:44.534743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:44.534707 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:44.698676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:44.698622 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:44.884180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:44.884070 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:45.092503 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:45.092465 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:45.256769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:45.256642 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:45.458915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:45.458872 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:45.663088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:45.662992 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:46.530880 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:46.530843 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:46.837170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:46.837085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:47.273255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:47.273228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:48.382199 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:48.382160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:50.082455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:50.082385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:51.654573 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:51.654538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:53.053513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:53.053482 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:53.214261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:53.214214 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:53.370004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:53.369920 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:55.039376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:55.039341 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:56.157966 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:32:56.157924 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:32:56.686979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:56.686933 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:58.253766 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:58.253729 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:32:59.856506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:32:59.856412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:01.430129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:01.430091 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:02.941008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:02.940962 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:04.586777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:04.586745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:06.173989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:06.173956 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:07.834402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:07.834366 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:09.268725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:09.268691 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:10.784653 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:10.784619 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:12.486082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:12.486044 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:14.056904 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:14.056867 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:15.685187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:15.685152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:17.372010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:17.371961 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:17.540217 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:17.540175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:18.974078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:18.974042 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:20.496976 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:20.496931 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:22.184502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:22.184457 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:23.797684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:23.797650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:25.481375 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:25.481334 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:26.158633 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:33:26.158597 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:33:27.218964 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:27.218930 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:28.947129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:28.947085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:30.578741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:30.578677 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:32.170894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:32.170853 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:33.783289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:33.783239 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:35.353102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:35.353066 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:36.955595 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:36.955555 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:38.437195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:38.437166 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:39.952735 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:39.952697 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:41.547988 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:41.547949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:43.077889 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:43.077833 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:44.592897 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:44.592854 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:46.156155 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:46.156118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:47.138394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:47.138359 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:47.304873 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:47.304832 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:47.910498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:47.910462 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:49.386458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:49.386416 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:51.033947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:51.033896 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:52.597526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:52.597478 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:54.109090 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:54.109055 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:55.714683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:55.714638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:56.159727 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:33:56.159689 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:33:57.337205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:57.337171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:33:58.973753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:33:58.973711 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:00.594991 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:00.594957 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:02.100179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:02.100132 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:02.592727 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:02.592668 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:02.740748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:02.740713 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:03.269228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:03.269190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:03.415970 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:03.415934 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:03.776310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:03.776273 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:05.281923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:05.281879 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:06.853690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:06.853631 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:08.320126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:08.320085 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:09.200593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:09.200552 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:09.794554 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:09.794522 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:11.357024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:11.356990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:11.702954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:11.702847 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:12.969605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:12.969518 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:14.795238 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:14.795204 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:16.544016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:16.543977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:17.420171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:17.420141 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:34:17.446473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:17.446405 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:34:17.664177 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:17.664145 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:18.277631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:18.277597 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:19.882228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:19.882190 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:21.617390 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:21.617357 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:23.171774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:23.171740 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:24.735677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:24.735638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:26.157904 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:34:26.157862 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:34:26.346389 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:26.346356 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:27.852179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:27.852147 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:29.422997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:29.422904 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:31.070718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:31.070689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:32.690574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:32.690515 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:34.380818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:34.380784 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:36.071008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:36.070972 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:37.631853 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:37.631823 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:39.289972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:39.289935 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:40.246493 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:40.246446 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb"] Nov 03 17:34:40.308796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:40.308754 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:40.976277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:40.976236 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:42.548649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:42.548612 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:44.067303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:44.067260 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:45.655118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:45.655082 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:47.188780 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:47.188749 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:47.568325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:47.568286 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n"] Nov 03 17:34:47.730448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:47.730347 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:48.984736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:48.984688 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:50.625237 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:50.625193 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:50.775449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:50.772867 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:51.042791 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:51.042761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:52.232437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:52.232391 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:53.749498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:53.749400 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:55.170441 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.165833 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg"] Nov 03 17:34:55.233356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.233317 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:55.286662 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.286621 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" podUID="ac86edc8-1098-4b62-a332-0a9d7865c52f" containerName="main" containerID="cri-o://114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00" gracePeriod=45 Nov 03 17:34:55.383515 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.383408 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:55.604946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.604913 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:34:55.680029 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.679988 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k\" (UniqueName: \"kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k\") pod \"ac86edc8-1098-4b62-a332-0a9d7865c52f\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " Nov 03 17:34:55.680231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680047 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm\") pod \"ac86edc8-1098-4b62-a332-0a9d7865c52f\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " Nov 03 17:34:55.680231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680137 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home\") pod \"ac86edc8-1098-4b62-a332-0a9d7865c52f\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " Nov 03 17:34:55.680231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680206 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache\") pod \"ac86edc8-1098-4b62-a332-0a9d7865c52f\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " Nov 03 17:34:55.680447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680349 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir\") pod \"ac86edc8-1098-4b62-a332-0a9d7865c52f\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " Nov 03 17:34:55.680513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680437 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home" pod "ac86edc8-1098-4b62-a332-0a9d7865c52f" (UID: "ac86edc8-1098-4b62-a332-0a9d7865c52f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:34:55.680513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680466 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs\") pod \"ac86edc8-1098-4b62-a332-0a9d7865c52f\" (UID: \"ac86edc8-1098-4b62-a332-0a9d7865c52f\") " Nov 03 17:34:55.680664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680614 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache" pod "ac86edc8-1098-4b62-a332-0a9d7865c52f" (UID: "ac86edc8-1098-4b62-a332-0a9d7865c52f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:34:55.680664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680643 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir" pod "ac86edc8-1098-4b62-a332-0a9d7865c52f" (UID: "ac86edc8-1098-4b62-a332-0a9d7865c52f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:34:55.682724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680936 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:55.682724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680957 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:55.682724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.680979 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:55.683105 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.683074 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k" pod "ac86edc8-1098-4b62-a332-0a9d7865c52f" (UID: "ac86edc8-1098-4b62-a332-0a9d7865c52f"). InnerVolumeSpecName "kube-api-access-9bh6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:55.683329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.683303 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm" pod "ac86edc8-1098-4b62-a332-0a9d7865c52f" (UID: "ac86edc8-1098-4b62-a332-0a9d7865c52f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:34:55.683832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.683799 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs" pod "ac86edc8-1098-4b62-a332-0a9d7865c52f" (UID: "ac86edc8-1098-4b62-a332-0a9d7865c52f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:55.689465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.688074 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:55.715403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.715369 4539 generic.go:356] "Generic (PLEG): container finished" podID="ac86edc8-1098-4b62-a332-0a9d7865c52f" containerID="114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00" exitCode=0 Nov 03 17:34:55.715619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.715470 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" Nov 03 17:34:55.715619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.715468 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" event={"ID":"ac86edc8-1098-4b62-a332-0a9d7865c52f","Type":"ContainerDied","Data":"114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00"} Nov 03 17:34:55.715619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.715521 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb" event={"ID":"ac86edc8-1098-4b62-a332-0a9d7865c52f","Type":"ContainerDied","Data":"3d59caecabbe283f13d2853097ae737542ea7ae92377f0b3b6e307c32e1b5632"} Nov 03 17:34:55.715619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.715544 4539 scope.go:122] "RemoveContainer" containerID="114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00" Nov 03 17:34:55.738936 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.738913 4539 scope.go:122] "RemoveContainer" containerID="114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00" Nov 03 17:34:55.739316 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:34:55.739292 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00\": container with ID starting with 114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00 not found: ID does not exist" containerID="114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00" Nov 03 17:34:55.739496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.739328 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00"} err="failed to get container status \"114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00\": rpc error: code = NotFound desc = could not find container \"114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00\": container with ID starting with 114a49f24453fb1d1c31e0ca5ba4e54bbc780de1e283d4b8b5cbc2e466c56c00 not found: ID does not exist" Nov 03 17:34:55.771218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.769851 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb"] Nov 03 17:34:55.775602 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.775566 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/rate-test-model-5a5ab1-kserve-d95d4cc7-jwjnb"] Nov 03 17:34:55.782572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.782540 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac86edc8-1098-4b62-a332-0a9d7865c52f-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:55.782572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.782567 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bh6k\" (UniqueName: \"kubernetes.io/projected/ac86edc8-1098-4b62-a332-0a9d7865c52f-kube-api-access-9bh6k\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:55.782779 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:55.782591 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac86edc8-1098-4b62-a332-0a9d7865c52f-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:56.158310 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:34:56.158271 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:34:56.469834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:56.469742 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac86edc8-1098-4b62-a332-0a9d7865c52f" path="/var/lib/kubelet/pods/ac86edc8-1098-4b62-a332-0a9d7865c52f/volumes" Nov 03 17:34:56.936644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:56.936612 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:57.209836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:57.209757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:57.383644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:57.383609 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:58.550763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.550729 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:58.588680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.588638 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz"] Nov 03 17:34:58.588938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.588905 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" podUID="23ce3998-66ad-4611-a897-911da8bb2194" containerName="maas-api" containerID="cri-o://ceacd04da6b58a0e689b0658487cb8c68bb6b9ec16b79c71bbe53c3b2df8e64e" gracePeriod=30 Nov 03 17:34:58.709808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.709781 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:58.761356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.761319 4539 generic.go:356] "Generic (PLEG): container finished" podID="23ce3998-66ad-4611-a897-911da8bb2194" containerID="ceacd04da6b58a0e689b0658487cb8c68bb6b9ec16b79c71bbe53c3b2df8e64e" exitCode=0 Nov 03 17:34:58.761579 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.761555 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" event={"ID":"23ce3998-66ad-4611-a897-911da8bb2194","Type":"ContainerDied","Data":"ceacd04da6b58a0e689b0658487cb8c68bb6b9ec16b79c71bbe53c3b2df8e64e"} Nov 03 17:34:58.952218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:58.952188 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:34:59.046454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.046222 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs\") pod \"23ce3998-66ad-4611-a897-911da8bb2194\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " Nov 03 17:34:59.046454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.046285 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls\") pod \"23ce3998-66ad-4611-a897-911da8bb2194\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " Nov 03 17:34:59.046454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.046370 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca\") pod \"23ce3998-66ad-4611-a897-911da8bb2194\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " Nov 03 17:34:59.046814 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.046512 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs\" (UniqueName: \"kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs\") pod \"23ce3998-66ad-4611-a897-911da8bb2194\" (UID: \"23ce3998-66ad-4611-a897-911da8bb2194\") " Nov 03 17:34:59.051454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.048447 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca" pod "23ce3998-66ad-4611-a897-911da8bb2194" (UID: "23ce3998-66ad-4611-a897-911da8bb2194"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:59.051454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.050596 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs" pod "23ce3998-66ad-4611-a897-911da8bb2194" (UID: "23ce3998-66ad-4611-a897-911da8bb2194"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:59.057746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.057597 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs" pod "23ce3998-66ad-4611-a897-911da8bb2194" (UID: "23ce3998-66ad-4611-a897-911da8bb2194"). InnerVolumeSpecName "kube-api-access-d77xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:59.062482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.057982 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls" pod "23ce3998-66ad-4611-a897-911da8bb2194" (UID: "23ce3998-66ad-4611-a897-911da8bb2194"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:59.148020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.147986 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d77xs\" (UniqueName: \"kubernetes.io/projected/23ce3998-66ad-4611-a897-911da8bb2194-kube-api-access-d77xs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:59.148020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.148018 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:59.148020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.148027 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ce3998-66ad-4611-a897-911da8bb2194-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:59.148304 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.148037 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ce3998-66ad-4611-a897-911da8bb2194-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:34:59.372351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.371390 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:34:59.773120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.773089 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" event={"ID":"23ce3998-66ad-4611-a897-911da8bb2194","Type":"ContainerDied","Data":"7674b68b948ae544c4742c3c8b8405a6c2973307f270d8b8bcdb33f958d9be14"} Nov 03 17:34:59.773695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.773148 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz" Nov 03 17:34:59.773695 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.773174 4539 scope.go:122] "RemoveContainer" containerID="ceacd04da6b58a0e689b0658487cb8c68bb6b9ec16b79c71bbe53c3b2df8e64e" Nov 03 17:34:59.825878 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.825842 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz"] Nov 03 17:34:59.837908 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:34:59.837873 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-da409b-5fc567cbdb-vn7gz"] Nov 03 17:35:00.470718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:00.470680 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ce3998-66ad-4611-a897-911da8bb2194" path="/var/lib/kubelet/pods/23ce3998-66ad-4611-a897-911da8bb2194/volumes" Nov 03 17:35:00.874074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:00.874042 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:02.240320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:02.240283 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:02.561290 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:02.561245 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:02.596053 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:02.596006 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" podUID="d1931cc8-edd5-401a-9589-61d65be8fdcf" containerName="main" containerID="cri-o://ea7ddc7ca00f4ba44b17b2de120a21500c945a087b23b542068f613490629afd" gracePeriod=45 Nov 03 17:35:02.798326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:02.798284 4539 generic.go:356] "Generic (PLEG): container finished" podID="d1931cc8-edd5-401a-9589-61d65be8fdcf" containerID="ea7ddc7ca00f4ba44b17b2de120a21500c945a087b23b542068f613490629afd" exitCode=0 Nov 03 17:35:02.798530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:02.798350 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" event={"ID":"d1931cc8-edd5-401a-9589-61d65be8fdcf","Type":"ContainerDied","Data":"ea7ddc7ca00f4ba44b17b2de120a21500c945a087b23b542068f613490629afd"} Nov 03 17:35:02.879111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:02.879084 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:35:03.002689 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.002657 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home\") pod \"d1931cc8-edd5-401a-9589-61d65be8fdcf\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " Nov 03 17:35:03.002902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.002755 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm\") pod \"d1931cc8-edd5-401a-9589-61d65be8fdcf\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " Nov 03 17:35:03.002902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.002892 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache\") pod \"d1931cc8-edd5-401a-9589-61d65be8fdcf\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " Nov 03 17:35:03.003035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.002934 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs\") pod \"d1931cc8-edd5-401a-9589-61d65be8fdcf\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " Nov 03 17:35:03.003035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.002946 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home" pod "d1931cc8-edd5-401a-9589-61d65be8fdcf" (UID: "d1931cc8-edd5-401a-9589-61d65be8fdcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:03.003035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.002959 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms\" (UniqueName: \"kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms\") pod \"d1931cc8-edd5-401a-9589-61d65be8fdcf\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " Nov 03 17:35:03.003231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.003053 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir\") pod \"d1931cc8-edd5-401a-9589-61d65be8fdcf\" (UID: \"d1931cc8-edd5-401a-9589-61d65be8fdcf\") " Nov 03 17:35:03.003231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.003189 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache" pod "d1931cc8-edd5-401a-9589-61d65be8fdcf" (UID: "d1931cc8-edd5-401a-9589-61d65be8fdcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:03.003337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.003314 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir" pod "d1931cc8-edd5-401a-9589-61d65be8fdcf" (UID: "d1931cc8-edd5-401a-9589-61d65be8fdcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:03.003898 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.003872 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:03.004041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.003902 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:03.004041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.003917 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:03.005313 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.005289 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm" pod "d1931cc8-edd5-401a-9589-61d65be8fdcf" (UID: "d1931cc8-edd5-401a-9589-61d65be8fdcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:03.005458 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.005359 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs" pod "d1931cc8-edd5-401a-9589-61d65be8fdcf" (UID: "d1931cc8-edd5-401a-9589-61d65be8fdcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:03.005521 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.005481 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms" pod "d1931cc8-edd5-401a-9589-61d65be8fdcf" (UID: "d1931cc8-edd5-401a-9589-61d65be8fdcf"). InnerVolumeSpecName "kube-api-access-rdvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:03.104969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.104876 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1931cc8-edd5-401a-9589-61d65be8fdcf-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:03.104969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.104915 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1931cc8-edd5-401a-9589-61d65be8fdcf-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:03.104969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.104929 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvms\" (UniqueName: \"kubernetes.io/projected/d1931cc8-edd5-401a-9589-61d65be8fdcf-kube-api-access-rdvms\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:03.408930 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.408845 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:03.609661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.609622 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:03.807318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.807281 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" Nov 03 17:35:03.807548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.807290 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n" event={"ID":"d1931cc8-edd5-401a-9589-61d65be8fdcf","Type":"ContainerDied","Data":"52bd7b730e41070f81d733099fdabaf2208a19c2f9f651b5d900a87bce4d2c36"} Nov 03 17:35:03.807548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.807418 4539 scope.go:122] "RemoveContainer" containerID="ea7ddc7ca00f4ba44b17b2de120a21500c945a087b23b542068f613490629afd" Nov 03 17:35:03.906151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.902263 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n"] Nov 03 17:35:03.912011 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:03.911967 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/rate-test-model-3b6f35-kserve-67cc487fc-z5f5n"] Nov 03 17:35:04.292568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:04.292533 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:04.469534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:04.469497 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1931cc8-edd5-401a-9589-61d65be8fdcf" path="/var/lib/kubelet/pods/d1931cc8-edd5-401a-9589-61d65be8fdcf/volumes" Nov 03 17:35:05.301627 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:05.301569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:05.474238 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:05.474199 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:06.168827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:06.168790 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:06.318102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:06.318072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:06.566334 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:06.566296 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj"] Nov 03 17:35:06.566848 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:06.566582 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" containerID="cri-o://06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9" gracePeriod=30 Nov 03 17:35:06.638976 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:06.638941 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:07.369121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:07.369093 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:07.967900 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:07.967866 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:08.416151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:08.416106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:09.571045 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:09.571013 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:10.198670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.198633 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" podUID="54fe1cf4-c205-43e8-b58f-8df0023999c0" containerName="main" containerID="cri-o://645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b" gracePeriod=45 Nov 03 17:35:10.321834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.321791 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" podUID="54fe1cf4-c205-43e8-b58f-8df0023999c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.206:8000/ready\": dial tcp 10.133.0.206:8000: connect: connection refused" Nov 03 17:35:10.466300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.466231 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:35:10.520019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.519990 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir\") pod \"54fe1cf4-c205-43e8-b58f-8df0023999c0\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " Nov 03 17:35:10.520210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.520095 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs\") pod \"54fe1cf4-c205-43e8-b58f-8df0023999c0\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " Nov 03 17:35:10.520210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.520182 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home\") pod \"54fe1cf4-c205-43e8-b58f-8df0023999c0\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " Nov 03 17:35:10.520348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.520211 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm\") pod \"54fe1cf4-c205-43e8-b58f-8df0023999c0\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " Nov 03 17:35:10.520348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.520285 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache\") pod \"54fe1cf4-c205-43e8-b58f-8df0023999c0\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " Nov 03 17:35:10.520488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.520399 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn\" (UniqueName: \"kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn\") pod \"54fe1cf4-c205-43e8-b58f-8df0023999c0\" (UID: \"54fe1cf4-c205-43e8-b58f-8df0023999c0\") " Nov 03 17:35:10.521861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.521583 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache" pod "54fe1cf4-c205-43e8-b58f-8df0023999c0" (UID: "54fe1cf4-c205-43e8-b58f-8df0023999c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:10.522233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.522189 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home" pod "54fe1cf4-c205-43e8-b58f-8df0023999c0" (UID: "54fe1cf4-c205-43e8-b58f-8df0023999c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:10.523667 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.522742 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir" pod "54fe1cf4-c205-43e8-b58f-8df0023999c0" (UID: "54fe1cf4-c205-43e8-b58f-8df0023999c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:10.525468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.524404 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs" pod "54fe1cf4-c205-43e8-b58f-8df0023999c0" (UID: "54fe1cf4-c205-43e8-b58f-8df0023999c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:10.525468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.524569 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn" pod "54fe1cf4-c205-43e8-b58f-8df0023999c0" (UID: "54fe1cf4-c205-43e8-b58f-8df0023999c0"). InnerVolumeSpecName "kube-api-access-zkqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:10.525468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.524670 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm" pod "54fe1cf4-c205-43e8-b58f-8df0023999c0" (UID: "54fe1cf4-c205-43e8-b58f-8df0023999c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:10.622687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.622647 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:10.622687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.622683 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:10.622687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.622691 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:10.623175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.622702 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqgn\" (UniqueName: \"kubernetes.io/projected/54fe1cf4-c205-43e8-b58f-8df0023999c0-kube-api-access-zkqgn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:10.623175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.622711 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe1cf4-c205-43e8-b58f-8df0023999c0-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:10.623175 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.622719 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe1cf4-c205-43e8-b58f-8df0023999c0-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:10.864187 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.864147 4539 generic.go:356] "Generic (PLEG): container finished" podID="54fe1cf4-c205-43e8-b58f-8df0023999c0" containerID="645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b" exitCode=0 Nov 03 17:35:10.864395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.864206 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" Nov 03 17:35:10.864395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.864227 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" event={"ID":"54fe1cf4-c205-43e8-b58f-8df0023999c0","Type":"ContainerDied","Data":"645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b"} Nov 03 17:35:10.864395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.864271 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg" event={"ID":"54fe1cf4-c205-43e8-b58f-8df0023999c0","Type":"ContainerDied","Data":"91090e3f5c6248976bdb83b8420697891fe67e33f7b1f2bdd3d03b7942e14b9a"} Nov 03 17:35:10.864395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.864289 4539 scope.go:122] "RemoveContainer" containerID="645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b" Nov 03 17:35:10.875785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.875768 4539 scope.go:122] "RemoveContainer" containerID="645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b" Nov 03 17:35:10.876042 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:10.876027 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b\": container with ID starting with 645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b not found: ID does not exist" containerID="645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b" Nov 03 17:35:10.876085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.876050 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b"} err="failed to get container status \"645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b\": rpc error: code = NotFound desc = could not find container \"645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b\": container with ID starting with 645a94246b9f1bc24e56b07b534258661e1295c940bfe50aec29bc10ded5ae4b not found: ID does not exist" Nov 03 17:35:10.917355 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.917324 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg"] Nov 03 17:35:10.922688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:10.922656 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c6f303/rate-test-model-c6f303-kserve-86d777466f-vdwrg"] Nov 03 17:35:11.368349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.368317 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:11.839323 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.839296 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:35:11.875263 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.875230 4539 generic.go:356] "Generic (PLEG): container finished" podID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerID="06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9" exitCode=0 Nov 03 17:35:11.875481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.875282 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" Nov 03 17:35:11.875481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.875312 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" event={"ID":"6b9f8004-34c4-49ab-b51b-4c050c3b8b29","Type":"ContainerDied","Data":"06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9"} Nov 03 17:35:11.875481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.875378 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj" event={"ID":"6b9f8004-34c4-49ab-b51b-4c050c3b8b29","Type":"ContainerDied","Data":"3f6806deb776284b7af08825c328dcd1135822b00c1e66ed1521e5a9c52116b3"} Nov 03 17:35:11.875481 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.875412 4539 scope.go:122] "RemoveContainer" containerID="06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9" Nov 03 17:35:11.887010 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.886987 4539 scope.go:122] "RemoveContainer" containerID="9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550" Nov 03 17:35:11.898129 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.898101 4539 scope.go:122] "RemoveContainer" containerID="06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9" Nov 03 17:35:11.898491 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:11.898461 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9\": container with ID starting with 06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9 not found: ID does not exist" containerID="06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9" Nov 03 17:35:11.898826 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.898501 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9"} err="failed to get container status \"06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9\": rpc error: code = NotFound desc = could not find container \"06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9\": container with ID starting with 06b363313176747c38d14355be16d63d3303f04879cffbe02f696d4f5a9813e9 not found: ID does not exist" Nov 03 17:35:11.898826 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.898524 4539 scope.go:122] "RemoveContainer" containerID="9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550" Nov 03 17:35:11.898975 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:11.898861 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550\": container with ID starting with 9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550 not found: ID does not exist" containerID="9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550" Nov 03 17:35:11.898975 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.898900 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550"} err="failed to get container status \"9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550\": rpc error: code = NotFound desc = could not find container \"9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550\": container with ID starting with 9aaeb7937f3588ee4d7686c9766b83e4caefd4fa7dd670c198e6eb328d683550 not found: ID does not exist" Nov 03 17:35:11.938599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938513 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.938751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938611 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.938751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938683 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.938751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938725 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.938941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938790 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.938941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938850 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.938941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938933 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.939097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938972 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:11.939097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.938984 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:11.939097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.939006 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.939097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.939041 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert\") pod \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\" (UID: \"6b9f8004-34c4-49ab-b51b-4c050c3b8b29\") " Nov 03 17:35:11.939547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.939522 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:11.939624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.939552 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:11.939728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.939705 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:11.939893 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.939872 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:11.941151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.940722 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:11.942722 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.942692 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:11.942961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.942931 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "kube-api-access-h64nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:11.943071 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.942969 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:11.943336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:11.943306 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token" pod "6b9f8004-34c4-49ab-b51b-4c050c3b8b29" (UID: "6b9f8004-34c4-49ab-b51b-4c050c3b8b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:12.041018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.040985 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.041018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.041015 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.041018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.041026 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.041272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.041037 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64nh\" (UniqueName: \"kubernetes.io/projected/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-kube-api-access-h64nh\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.041272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.041048 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.041272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.041056 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.041272 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.041063 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b9f8004-34c4-49ab-b51b-4c050c3b8b29-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:12.223668 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.223585 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj"] Nov 03 17:35:12.235611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.235577 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-k9zhj"] Nov 03 17:35:12.469737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.469701 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fe1cf4-c205-43e8-b58f-8df0023999c0" path="/var/lib/kubelet/pods/54fe1cf4-c205-43e8-b58f-8df0023999c0/volumes" Nov 03 17:35:12.470214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:12.470199 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" path="/var/lib/kubelet/pods/6b9f8004-34c4-49ab-b51b-4c050c3b8b29/volumes" Nov 03 17:35:13.058350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.058310 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11342-dd64fa"] Nov 03 17:35:13.059145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059124 4539 cpu_manager.go:414] "containerMap: removing container" podUID="23ce3998-66ad-4611-a897-911da8bb2194" containerName="maas-api" Nov 03 17:35:13.059145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059145 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ce3998-66ad-4611-a897-911da8bb2194" containerName="maas-api" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059166 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a139fc2b-2df8-44dd-9356-2cefc49d160d" containerName="maas-api" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059171 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a139fc2b-2df8-44dd-9356-2cefc49d160d" containerName="maas-api" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059179 4539 cpu_manager.go:414] "containerMap: removing container" podUID="54fe1cf4-c205-43e8-b58f-8df0023999c0" containerName="main" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059184 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fe1cf4-c205-43e8-b58f-8df0023999c0" containerName="main" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059199 4539 cpu_manager.go:414] "containerMap: removing container" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059205 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059221 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ac86edc8-1098-4b62-a332-0a9d7865c52f" containerName="main" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059228 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac86edc8-1098-4b62-a332-0a9d7865c52f" containerName="main" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059242 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d1931cc8-edd5-401a-9589-61d65be8fdcf" containerName="main" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059248 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1931cc8-edd5-401a-9589-61d65be8fdcf" containerName="main" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059258 4539 cpu_manager.go:414] "containerMap: removing container" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" Nov 03 17:35:13.059347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059265 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059411 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1931cc8-edd5-401a-9589-61d65be8fdcf" containerName="main" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059443 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059453 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c641e1c4-3301-4c83-85a1-f17596684ea7" containerName="istio-proxy" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059466 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ce3998-66ad-4611-a897-911da8bb2194" containerName="maas-api" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059475 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a139fc2b-2df8-44dd-9356-2cefc49d160d" containerName="maas-api" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059482 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fe1cf4-c205-43e8-b58f-8df0023999c0" containerName="main" Nov 03 17:35:13.059995 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.059495 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac86edc8-1098-4b62-a332-0a9d7865c52f" containerName="main" Nov 03 17:35:13.064259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.064241 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:13.069119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.069092 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:13.080002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.079976 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 17:35:13.084696 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.084667 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11342-dd64fa"] Nov 03 17:35:13.153652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.153603 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4f49\" (UniqueName: \"kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49\") pod \"test-curl-11342-dd64fa\" (UID: \"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66\") " pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:13.221787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.221689 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:13.255177 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.255135 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4f49\" (UniqueName: \"kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49\") pod \"test-curl-11342-dd64fa\" (UID: \"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66\") " pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:13.278558 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.278531 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4f49\" (UniqueName: \"kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49\") pod \"test-curl-11342-dd64fa\" (UID: \"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66\") " pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:13.378170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.378081 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:13.682294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.682261 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11342-dd64fa"] Nov 03 17:35:13.684160 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:35:13.684128 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04f6d5e_c670_4d8c_8f68_69b9bfd00e66.slice/crio-789f4e0352c3a4292974918becc1955238b39089d3ec5717e7029172ff5d3668 WatchSource:0}: Error finding container 789f4e0352c3a4292974918becc1955238b39089d3ec5717e7029172ff5d3668: Status 404 returned error can't find the container with id 789f4e0352c3a4292974918becc1955238b39089d3ec5717e7029172ff5d3668 Nov 03 17:35:13.892878 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.892840 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11342-dd64fa" event={"ID":"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66","Type":"ContainerStarted","Data":"66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e"} Nov 03 17:35:13.893061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.892891 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11342-dd64fa" event={"ID":"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66","Type":"ContainerStarted","Data":"789f4e0352c3a4292974918becc1955238b39089d3ec5717e7029172ff5d3668"} Nov 03 17:35:13.935634 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.935524 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11342-dd64fa" podStartSLOduration=0.93551129 podStartE2EDuration="935.51129ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:13.933762105 +0000 UTC m=+4857.627033126" watchObservedRunningTime="2026-11-03 17:35:13.93551129 +0000 UTC m=+4857.628782312" Nov 03 17:35:13.983631 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:13.983600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:14.730493 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.730457 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11342-dd64fa"] Nov 03 17:35:14.734541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.734513 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11342-dd64fa"] Nov 03 17:35:14.767032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.767002 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:14.878150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.878114 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11342-27034e"] Nov 03 17:35:14.878731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.878714 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" containerName="test-curl-11342-dd64fa" Nov 03 17:35:14.878837 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.878732 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" containerName="test-curl-11342-dd64fa" Nov 03 17:35:14.878897 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.878837 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9f8004-34c4-49ab-b51b-4c050c3b8b29" containerName="istio-proxy" Nov 03 17:35:14.878897 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.878852 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" containerName="test-curl-11342-dd64fa" Nov 03 17:35:14.882437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.882400 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:14.906356 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.906326 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11342-27034e"] Nov 03 17:35:14.976410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:14.976366 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmgq\" (UniqueName: \"kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq\") pod \"test-curl-11342-27034e\" (UID: \"7181cef8-9af4-472b-bda2-8829af32400a\") " pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:15.077506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.077467 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmgq\" (UniqueName: \"kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq\") pod \"test-curl-11342-27034e\" (UID: \"7181cef8-9af4-472b-bda2-8829af32400a\") " pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:15.100624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.100589 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmgq\" (UniqueName: \"kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq\") pod \"test-curl-11342-27034e\" (UID: \"7181cef8-9af4-472b-bda2-8829af32400a\") " pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:15.194858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.194827 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:15.495988 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.495960 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11342-27034e"] Nov 03 17:35:15.497921 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:35:15.497899 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7181cef8_9af4_472b_bda2_8829af32400a.slice/crio-29d318706a8b0b45d70597d751996fb3f4e9279b5fb8cee85c318377ad0e41ba WatchSource:0}: Error finding container 29d318706a8b0b45d70597d751996fb3f4e9279b5fb8cee85c318377ad0e41ba: Status 404 returned error can't find the container with id 29d318706a8b0b45d70597d751996fb3f4e9279b5fb8cee85c318377ad0e41ba Nov 03 17:35:15.909046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.908996 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11342-27034e" event={"ID":"7181cef8-9af4-472b-bda2-8829af32400a","Type":"ContainerStarted","Data":"9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540"} Nov 03 17:35:15.909046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.909048 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11342-27034e" event={"ID":"7181cef8-9af4-472b-bda2-8829af32400a","Type":"ContainerStarted","Data":"29d318706a8b0b45d70597d751996fb3f4e9279b5fb8cee85c318377ad0e41ba"} Nov 03 17:35:15.909638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.909295 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11342-dd64fa" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" containerName="test-curl-11342-dd64fa" containerID="cri-o://66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e" gracePeriod=2 Nov 03 17:35:15.948208 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:15.948147 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11342-27034e" podStartSLOduration=1.948130757 podStartE2EDuration="1.948130757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:15.947031978 +0000 UTC m=+4859.640302994" watchObservedRunningTime="2026-11-03 17:35:15.948130757 +0000 UTC m=+4859.641401778" Nov 03 17:35:15.960345 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:15.960306 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11342-dd64fa\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:16.075694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:16.075659 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:16.479817 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:16.479778 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11342-dd64fa\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:16.747705 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:16.747617 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:16.785483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:16.785410 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11342-27034e"] Nov 03 17:35:16.790081 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:16.790055 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11342-27034e"] Nov 03 17:35:17.815437 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:17.815398 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:17.924335 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:17.924266 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11342-27034e" podUID="7181cef8-9af4-472b-bda2-8829af32400a" containerName="test-curl-11342-27034e" containerID="cri-o://9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540" gracePeriod=2 Nov 03 17:35:17.937883 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:17.937829 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11342-27034e\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="7181cef8-9af4-472b-bda2-8829af32400a" pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:18.117078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.117043 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:18.130673 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:18.130628 4539 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11342-27034e\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="7181cef8-9af4-472b-bda2-8829af32400a" pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:18.215990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.215958 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49\" (UniqueName: \"kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49\") pod \"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66\" (UID: \"b04f6d5e-c670-4d8c-8f68-69b9bfd00e66\") " Nov 03 17:35:18.218368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.218333 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49" pod "b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" (UID: "b04f6d5e-c670-4d8c-8f68-69b9bfd00e66"). InnerVolumeSpecName "kube-api-access-n4f49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:18.317955 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.317913 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4f49\" (UniqueName: \"kubernetes.io/projected/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66-kube-api-access-n4f49\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:18.374716 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.374610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:18.469007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.468968 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" path="/var/lib/kubelet/pods/b04f6d5e-c670-4d8c-8f68-69b9bfd00e66/volumes" Nov 03 17:35:18.934742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.934707 4539 generic.go:356] "Generic (PLEG): container finished" podID="b04f6d5e-c670-4d8c-8f68-69b9bfd00e66" containerID="66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e" exitCode=137 Nov 03 17:35:18.935283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.934774 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-dd64fa" Nov 03 17:35:18.935283 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.934821 4539 scope.go:122] "RemoveContainer" containerID="66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e" Nov 03 17:35:18.947104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.947072 4539 scope.go:122] "RemoveContainer" containerID="66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e" Nov 03 17:35:18.947546 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:18.947518 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e\": container with ID starting with 66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e not found: ID does not exist" containerID="66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e" Nov 03 17:35:18.947655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:18.947561 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e"} err="failed to get container status \"66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e\": rpc error: code = NotFound desc = could not find container \"66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e\": container with ID starting with 66c11878f04df57ae9c2d07a5346c17c2d0083ee5af48bed8c0380268f388b2e not found: ID does not exist" Nov 03 17:35:19.967483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:19.967412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:20.168599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.168564 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:20.340466 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.340391 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq\" (UniqueName: \"kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq\") pod \"7181cef8-9af4-472b-bda2-8829af32400a\" (UID: \"7181cef8-9af4-472b-bda2-8829af32400a\") " Nov 03 17:35:20.342802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.342771 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq" pod "7181cef8-9af4-472b-bda2-8829af32400a" (UID: "7181cef8-9af4-472b-bda2-8829af32400a"). InnerVolumeSpecName "kube-api-access-4mmgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:20.442022 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.441978 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mmgq\" (UniqueName: \"kubernetes.io/projected/7181cef8-9af4-472b-bda2-8829af32400a-kube-api-access-4mmgq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:20.471275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.471237 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7181cef8-9af4-472b-bda2-8829af32400a" path="/var/lib/kubelet/pods/7181cef8-9af4-472b-bda2-8829af32400a/volumes" Nov 03 17:35:20.953405 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.953371 4539 generic.go:356] "Generic (PLEG): container finished" podID="7181cef8-9af4-472b-bda2-8829af32400a" containerID="9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540" exitCode=137 Nov 03 17:35:20.953655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.953444 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11342-27034e" Nov 03 17:35:20.953655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.953471 4539 scope.go:122] "RemoveContainer" containerID="9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540" Nov 03 17:35:20.966031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.966006 4539 scope.go:122] "RemoveContainer" containerID="9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540" Nov 03 17:35:20.966408 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:20.966384 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540\": container with ID starting with 9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540 not found: ID does not exist" containerID="9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540" Nov 03 17:35:20.966512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:20.966418 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540"} err="failed to get container status \"9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540\": rpc error: code = NotFound desc = could not find container \"9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540\": container with ID starting with 9e5dff07204320da49e9993c780d96e8c12e4970ea3923349955d1b22d580540 not found: ID does not exist" Nov 03 17:35:21.288872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:21.288837 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:21.813688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:21.813624 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:23.389527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:23.389485 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:24.249536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:24.249294 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:24.419987 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:24.419956 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:25.017567 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:25.017538 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:25.240374 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:25.240340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:25.450223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:25.450184 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:26.158601 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:26.158550 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:35:26.261365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:26.261324 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:26.776403 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:26.776364 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:28.345032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:28.344988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:29.826860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:29.826820 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd"] Nov 03 17:35:29.846965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:29.827113 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" podUID="d3f63229-5b65-4992-ba0c-69baf3a814a4" containerName="maas-api" containerID="cri-o://17369f2360a458b78cab3c15b6c76160f7c00e8656f8d205c7d4df20bec5e875" gracePeriod=30 Nov 03 17:35:29.920041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:29.920007 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:30.017622 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.017584 4539 generic.go:356] "Generic (PLEG): container finished" podID="d3f63229-5b65-4992-ba0c-69baf3a814a4" containerID="17369f2360a458b78cab3c15b6c76160f7c00e8656f8d205c7d4df20bec5e875" exitCode=0 Nov 03 17:35:30.017832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.017652 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" event={"ID":"d3f63229-5b65-4992-ba0c-69baf3a814a4","Type":"ContainerDied","Data":"17369f2360a458b78cab3c15b6c76160f7c00e8656f8d205c7d4df20bec5e875"} Nov 03 17:35:30.209630 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.209598 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:35:30.289447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.288472 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs\") pod \"d3f63229-5b65-4992-ba0c-69baf3a814a4\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " Nov 03 17:35:30.289447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.288567 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca\") pod \"d3f63229-5b65-4992-ba0c-69baf3a814a4\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " Nov 03 17:35:30.289447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.288708 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9\" (UniqueName: \"kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9\") pod \"d3f63229-5b65-4992-ba0c-69baf3a814a4\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " Nov 03 17:35:30.289447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.288758 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls\") pod \"d3f63229-5b65-4992-ba0c-69baf3a814a4\" (UID: \"d3f63229-5b65-4992-ba0c-69baf3a814a4\") " Nov 03 17:35:30.292317 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.292251 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca" pod "d3f63229-5b65-4992-ba0c-69baf3a814a4" (UID: "d3f63229-5b65-4992-ba0c-69baf3a814a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:30.301025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.300961 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls" pod "d3f63229-5b65-4992-ba0c-69baf3a814a4" (UID: "d3f63229-5b65-4992-ba0c-69baf3a814a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:30.305091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.303051 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9" pod "d3f63229-5b65-4992-ba0c-69baf3a814a4" (UID: "d3f63229-5b65-4992-ba0c-69baf3a814a4"). InnerVolumeSpecName "kube-api-access-vfll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:30.305091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.303103 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs" pod "d3f63229-5b65-4992-ba0c-69baf3a814a4" (UID: "d3f63229-5b65-4992-ba0c-69baf3a814a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:30.391132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.391086 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfll9\" (UniqueName: \"kubernetes.io/projected/d3f63229-5b65-4992-ba0c-69baf3a814a4-kube-api-access-vfll9\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.391132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.391128 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.391132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.391141 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f63229-5b65-4992-ba0c-69baf3a814a4-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.391406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.391152 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f63229-5b65-4992-ba0c-69baf3a814a4-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.650027 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:30.649926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:31.027444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:31.027395 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" Nov 03 17:35:31.027992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:31.027452 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd" event={"ID":"d3f63229-5b65-4992-ba0c-69baf3a814a4","Type":"ContainerDied","Data":"f762d70b9ba033a976200633de5cb9a5546eb42765a934c7ae25c8a626d01c86"} Nov 03 17:35:31.027992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:31.027528 4539 scope.go:122] "RemoveContainer" containerID="17369f2360a458b78cab3c15b6c76160f7c00e8656f8d205c7d4df20bec5e875" Nov 03 17:35:31.079434 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:31.079381 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd"] Nov 03 17:35:31.091120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:31.091086 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e0e63d-55457756dd-74xjd"] Nov 03 17:35:32.253399 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:32.253365 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:32.469602 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:32.469566 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f63229-5b65-4992-ba0c-69baf3a814a4" path="/var/lib/kubelet/pods/d3f63229-5b65-4992-ba0c-69baf3a814a4/volumes" Nov 03 17:35:32.502573 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:32.502534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:33.208888 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:33.208846 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:33.998849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:33.998812 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:35.768460 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:35.768388 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:36.967799 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:36.967760 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:37.314797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:37.314762 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:37.597589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:37.597491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:37.885050 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:37.884944 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:38.215764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:38.215644 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk"] Nov 03 17:35:38.216322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:38.216289 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" containerID="cri-o://2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd" gracePeriod=30 Nov 03 17:35:38.304924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:38.304889 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:39.188500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:39.188462 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:39.392217 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:39.392181 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:39.577713 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:39.577682 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:39.728204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:39.728164 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:40.676255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:40.676219 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps"] Nov 03 17:35:40.676809 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:40.676445 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" podUID="3d620953-d68f-4c88-b01c-d31fc541dd04" containerName="maas-api" containerID="cri-o://3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955" gracePeriod=30 Nov 03 17:35:40.791179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:40.791142 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:40.900751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:40.900337 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:35:40.900751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:40.900462 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:35:41.018595 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.018555 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:35:41.099219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.099184 4539 generic.go:356] "Generic (PLEG): container finished" podID="3d620953-d68f-4c88-b01c-d31fc541dd04" containerID="3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955" exitCode=0 Nov 03 17:35:41.099409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.099237 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" Nov 03 17:35:41.099409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.099262 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" event={"ID":"3d620953-d68f-4c88-b01c-d31fc541dd04","Type":"ContainerDied","Data":"3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955"} Nov 03 17:35:41.099409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.099331 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps" event={"ID":"3d620953-d68f-4c88-b01c-d31fc541dd04","Type":"ContainerDied","Data":"91a8cd38f7bfb065569002898987b1b0589eff2e9bcce05ffba769ea76238ac1"} Nov 03 17:35:41.099409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.099366 4539 scope.go:122] "RemoveContainer" containerID="3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955" Nov 03 17:35:41.110034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.110008 4539 scope.go:122] "RemoveContainer" containerID="3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955" Nov 03 17:35:41.110414 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:41.110390 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955\": container with ID starting with 3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955 not found: ID does not exist" containerID="3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955" Nov 03 17:35:41.110547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.110465 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955"} err="failed to get container status \"3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955\": rpc error: code = NotFound desc = could not find container \"3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955\": container with ID starting with 3233ffeae1e7642f381bfe6dc877be453bb864eaea6867b6ef588752bcec1955 not found: ID does not exist" Nov 03 17:35:41.118948 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.118901 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca\") pod \"3d620953-d68f-4c88-b01c-d31fc541dd04\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " Nov 03 17:35:41.119128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.119027 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv\" (UniqueName: \"kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv\") pod \"3d620953-d68f-4c88-b01c-d31fc541dd04\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " Nov 03 17:35:41.119215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.119197 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs\") pod \"3d620953-d68f-4c88-b01c-d31fc541dd04\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " Nov 03 17:35:41.119382 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.119357 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca" pod "3d620953-d68f-4c88-b01c-d31fc541dd04" (UID: "3d620953-d68f-4c88-b01c-d31fc541dd04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:41.119823 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.119228 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls\") pod \"3d620953-d68f-4c88-b01c-d31fc541dd04\" (UID: \"3d620953-d68f-4c88-b01c-d31fc541dd04\") " Nov 03 17:35:41.120554 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.120535 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d620953-d68f-4c88-b01c-d31fc541dd04-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:41.122084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.122061 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv" pod "3d620953-d68f-4c88-b01c-d31fc541dd04" (UID: "3d620953-d68f-4c88-b01c-d31fc541dd04"). InnerVolumeSpecName "kube-api-access-556dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:41.122286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.122259 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls" pod "3d620953-d68f-4c88-b01c-d31fc541dd04" (UID: "3d620953-d68f-4c88-b01c-d31fc541dd04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:41.124332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.124309 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs" pod "3d620953-d68f-4c88-b01c-d31fc541dd04" (UID: "3d620953-d68f-4c88-b01c-d31fc541dd04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:41.243547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.235797 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556dv\" (UniqueName: \"kubernetes.io/projected/3d620953-d68f-4c88-b01c-d31fc541dd04-kube-api-access-556dv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:41.243547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.235838 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:41.243547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.235853 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d620953-d68f-4c88-b01c-d31fc541dd04-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:41.451154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.451109 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps"] Nov 03 17:35:41.464527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.464492 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ca9d7c-7f4b544db-r6jps"] Nov 03 17:35:41.528766 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.528731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:41.689778 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:41.689743 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:42.468584 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:42.468544 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d620953-d68f-4c88-b01c-d31fc541dd04" path="/var/lib/kubelet/pods/3d620953-d68f-4c88-b01c-d31fc541dd04/volumes" Nov 03 17:35:43.489891 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.489776 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:43.491868 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.491510 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:35:43.561484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561453 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.561728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561504 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.561728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561619 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.561864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561731 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.561864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561758 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.561864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561771 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:43.561864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561792 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.562076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561880 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.562076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561912 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.562076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561927 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:43.562076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.561969 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket\") pod \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\" (UID: \"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3\") " Nov 03 17:35:43.562289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.562118 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:43.562719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.562383 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:43.562719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.562392 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:43.563855 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.563827 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.563974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.563860 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.563974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.563874 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.563974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.563887 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.563974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.563900 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.566070 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.566043 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:43.566182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.566082 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "kube-api-access-dpmg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:43.566182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.566158 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:43.566761 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.566733 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy" pod "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" (UID: "bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:43.665211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.665167 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.665211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.665203 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.665211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.665212 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:43.665211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:43.665221 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmg9\" (UniqueName: \"kubernetes.io/projected/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3-kube-api-access-dpmg9\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:44.131357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.131323 4539 generic.go:356] "Generic (PLEG): container finished" podID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerID="2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd" exitCode=0 Nov 03 17:35:44.131611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.131376 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" Nov 03 17:35:44.131611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.131408 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" event={"ID":"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3","Type":"ContainerDied","Data":"2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd"} Nov 03 17:35:44.131611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.131506 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk" event={"ID":"bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3","Type":"ContainerDied","Data":"98562a362040a2b5b86c79f1be48b853c386dbad65f4bb38397dd5cb98e21603"} Nov 03 17:35:44.131611 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.131538 4539 scope.go:122] "RemoveContainer" containerID="2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd" Nov 03 17:35:44.144231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.144211 4539 scope.go:122] "RemoveContainer" containerID="6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d" Nov 03 17:35:44.158113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.158032 4539 scope.go:122] "RemoveContainer" containerID="2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd" Nov 03 17:35:44.158629 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:44.158602 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd\": container with ID starting with 2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd not found: ID does not exist" containerID="2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd" Nov 03 17:35:44.158748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.158636 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd"} err="failed to get container status \"2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd\": rpc error: code = NotFound desc = could not find container \"2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd\": container with ID starting with 2f13958edf6c762e1f369fcfc958506a5afcf80757a985213f6f13d87be4c7fd not found: ID does not exist" Nov 03 17:35:44.158748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.158656 4539 scope.go:122] "RemoveContainer" containerID="6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d" Nov 03 17:35:44.159054 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:44.159033 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d\": container with ID starting with 6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d not found: ID does not exist" containerID="6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d" Nov 03 17:35:44.159119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.159061 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d"} err="failed to get container status \"6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d\": rpc error: code = NotFound desc = could not find container \"6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d\": container with ID starting with 6627c7ada97bd0716cec15ca0e5693165e6328d4c318cf3aa8012c57035b048d not found: ID does not exist" Nov 03 17:35:44.184242 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.184190 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk"] Nov 03 17:35:44.196714 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.196682 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-tb4mk"] Nov 03 17:35:44.468917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.468825 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" path="/var/lib/kubelet/pods/bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3/volumes" Nov 03 17:35:44.731773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.731675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:44.876215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:44.876180 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:45.206985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:45.206949 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:45.785974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:45.785936 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:46.772496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:46.772411 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:47.387180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:47.386235 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:48.063214 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:48.063172 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:48.210556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:48.210497 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:48.709154 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:48.709118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:48.985320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:48.985179 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:49.063977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:49.063937 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48"] Nov 03 17:35:49.064258 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:49.064218 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" containerID="cri-o://d4a0c8cded6ec9b05a7df192de8882b48e76e0e6226fb22aee5e2488fc2195c9" gracePeriod=30 Nov 03 17:35:49.142357 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:49.142318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:49.875074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:49.875037 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:50.671866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:50.671829 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:52.399191 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:52.397291 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:53.201619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:53.201572 4539 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:35:53.201833 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:53.201663 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:35:53.752562 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:53.752526 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:53.896353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:53.896325 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:54.050754 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.050694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:54.250040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.249893 4539 generic.go:356] "Generic (PLEG): container finished" podID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerID="d4a0c8cded6ec9b05a7df192de8882b48e76e0e6226fb22aee5e2488fc2195c9" exitCode=0 Nov 03 17:35:54.250040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.249951 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" event={"ID":"f7b13c5a-b79f-40ef-98bc-b0606db9772e","Type":"ContainerDied","Data":"d4a0c8cded6ec9b05a7df192de8882b48e76e0e6226fb22aee5e2488fc2195c9"} Nov 03 17:35:54.250040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.250020 4539 scope.go:122] "RemoveContainer" containerID="834af19cd358133b3588d89c8166ae73ad57a42726fd3e397191dd79c4a81d17" Nov 03 17:35:54.350816 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.350793 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:35:54.413212 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413172 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413236 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413319 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413371 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413441 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413508 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413480 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413542 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413572 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.413773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413629 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token\") pod \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\" (UID: \"f7b13c5a-b79f-40ef-98bc-b0606db9772e\") " Nov 03 17:35:54.414015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.413992 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:54.415352 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.415086 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:54.415628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.415564 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:54.415804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.415768 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:54.415804 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.415777 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:54.418569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.418538 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:54.422450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.422405 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "kube-api-access-2j2ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:54.422849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.422814 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:54.423883 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.423852 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy" pod "f7b13c5a-b79f-40ef-98bc-b0606db9772e" (UID: "f7b13c5a-b79f-40ef-98bc-b0606db9772e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:54.448408 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.445012 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:54.514569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514535 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2ql\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-kube-api-access-2j2ql\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514569 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514582 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514594 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514605 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514616 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514626 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514637 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.514769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.514648 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b13c5a-b79f-40ef-98bc-b0606db9772e-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:35:54.944449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:54.942935 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:55.258164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.258064 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" Nov 03 17:35:55.258164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.258095 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48" event={"ID":"f7b13c5a-b79f-40ef-98bc-b0606db9772e","Type":"ContainerDied","Data":"6d6f67ed4b2c037bd6369894b30b1fb1456b841c5ed9f3c62930b4ad0a13008b"} Nov 03 17:35:55.258400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.258170 4539 scope.go:122] "RemoveContainer" containerID="d4a0c8cded6ec9b05a7df192de8882b48e76e0e6226fb22aee5e2488fc2195c9" Nov 03 17:35:55.322840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.322794 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48"] Nov 03 17:35:55.334915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.334874 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-cts48"] Nov 03 17:35:55.571140 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.571106 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:55.758958 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:55.758929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:56.166084 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:35:56.166044 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:35:56.196680 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:56.196650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:56.398345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:56.398306 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:56.469067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:56.468981 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" path="/var/lib/kubelet/pods/f7b13c5a-b79f-40ef-98bc-b0606db9772e/volumes" Nov 03 17:35:56.603008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:56.602975 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:57.590257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:57.590219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:35:59.200338 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:35:59.200240 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:00.163041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:00.162996 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79"] Nov 03 17:36:00.228735 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:00.228665 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:00.950796 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:00.950761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:02.590373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:02.590342 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:02.972084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:02.971996 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:03.133217 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:03.133172 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:03.234522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:03.234264 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk"] Nov 03 17:36:03.302759 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:03.302724 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:03.457183 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:03.457141 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:03.753766 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:03.753728 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:04.600502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:04.600454 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:04.739119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:04.739083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:06.084907 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:06.084871 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:07.572347 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:07.572311 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:09.083636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:09.083575 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:09.835193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:09.835160 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:10.376182 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.376145 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92"] Nov 03 17:36:10.376952 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.376468 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" podUID="400a9d3e-8c3a-4669-94a2-4642adbe2525" containerName="maas-api" containerID="cri-o://23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e" gracePeriod=30 Nov 03 17:36:10.473132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.473086 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:10.700482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.700449 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 17:36:10.815052 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.815019 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls\") pod \"400a9d3e-8c3a-4669-94a2-4642adbe2525\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " Nov 03 17:36:10.815281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.815092 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs\") pod \"400a9d3e-8c3a-4669-94a2-4642adbe2525\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " Nov 03 17:36:10.815281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.815265 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca\") pod \"400a9d3e-8c3a-4669-94a2-4642adbe2525\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " Nov 03 17:36:10.815414 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.815324 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq\" (UniqueName: \"kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq\") pod \"400a9d3e-8c3a-4669-94a2-4642adbe2525\" (UID: \"400a9d3e-8c3a-4669-94a2-4642adbe2525\") " Nov 03 17:36:10.816058 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.816026 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca" pod "400a9d3e-8c3a-4669-94a2-4642adbe2525" (UID: "400a9d3e-8c3a-4669-94a2-4642adbe2525"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:10.818259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.818227 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls" pod "400a9d3e-8c3a-4669-94a2-4642adbe2525" (UID: "400a9d3e-8c3a-4669-94a2-4642adbe2525"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:10.818259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.818247 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs" pod "400a9d3e-8c3a-4669-94a2-4642adbe2525" (UID: "400a9d3e-8c3a-4669-94a2-4642adbe2525"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:10.818396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.818342 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq" pod "400a9d3e-8c3a-4669-94a2-4642adbe2525" (UID: "400a9d3e-8c3a-4669-94a2-4642adbe2525"). InnerVolumeSpecName "kube-api-access-w96jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:10.916886 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.916847 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400a9d3e-8c3a-4669-94a2-4642adbe2525-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:10.916886 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.916881 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96jq\" (UniqueName: \"kubernetes.io/projected/400a9d3e-8c3a-4669-94a2-4642adbe2525-kube-api-access-w96jq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:10.916886 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.916892 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:10.917161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:10.916902 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400a9d3e-8c3a-4669-94a2-4642adbe2525-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:11.078763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.078731 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:11.256222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.256191 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:11.404513 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.404377 4539 generic.go:356] "Generic (PLEG): container finished" podID="400a9d3e-8c3a-4669-94a2-4642adbe2525" containerID="23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e" exitCode=0 Nov 03 17:36:11.405002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.404520 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" event={"ID":"400a9d3e-8c3a-4669-94a2-4642adbe2525","Type":"ContainerDied","Data":"23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e"} Nov 03 17:36:11.405002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.404572 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" event={"ID":"400a9d3e-8c3a-4669-94a2-4642adbe2525","Type":"ContainerDied","Data":"a00d08b8ea271d919fea7a08258baffb0afcdc1f1bd47ec16651ab83ebeb1bf1"} Nov 03 17:36:11.405002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.404603 4539 scope.go:122] "RemoveContainer" containerID="23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e" Nov 03 17:36:11.405002 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.404770 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92" Nov 03 17:36:11.425044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.425020 4539 scope.go:122] "RemoveContainer" containerID="23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e" Nov 03 17:36:11.425440 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:11.425399 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e\": container with ID starting with 23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e not found: ID does not exist" containerID="23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e" Nov 03 17:36:11.425580 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.425452 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e"} err="failed to get container status \"23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e\": rpc error: code = NotFound desc = could not find container \"23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e\": container with ID starting with 23a8419a47a6f9b2101b6f53233312e62d25e6c6ab072d005932ea2bd5c51d1e not found: ID does not exist" Nov 03 17:36:11.458893 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.458857 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:11.459576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.459552 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92"] Nov 03 17:36:11.463625 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.463605 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3b6f35-56b7fd699-d5f92"] Nov 03 17:36:11.708231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:11.708125 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:12.188311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:12.188270 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:12.421289 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:12.421258 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:12.468991 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:12.468911 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400a9d3e-8c3a-4669-94a2-4642adbe2525" path="/var/lib/kubelet/pods/400a9d3e-8c3a-4669-94a2-4642adbe2525/volumes" Nov 03 17:36:12.845507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:12.845473 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t"] Nov 03 17:36:12.845742 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:12.845713 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" podUID="5c982958-c024-4297-a336-c09c6f42480b" containerName="maas-api" containerID="cri-o://a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701" gracePeriod=30 Nov 03 17:36:12.960186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:12.960149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:13.132468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.127720 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:13.195233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.195208 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 17:36:13.270107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.270069 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c"] Nov 03 17:36:13.277697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.270341 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" containerID="cri-o://96e6fd4363ceec7ff9187cacb19a72daead427afb21ac9da56964a6a01968db7" gracePeriod=30 Nov 03 17:36:13.351309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.351272 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:13.356093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.356063 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls\") pod \"5c982958-c024-4297-a336-c09c6f42480b\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " Nov 03 17:36:13.356280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.356119 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2\" (UniqueName: \"kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2\") pod \"5c982958-c024-4297-a336-c09c6f42480b\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " Nov 03 17:36:13.356280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.356156 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs\") pod \"5c982958-c024-4297-a336-c09c6f42480b\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " Nov 03 17:36:13.356280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.356192 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca\") pod \"5c982958-c024-4297-a336-c09c6f42480b\" (UID: \"5c982958-c024-4297-a336-c09c6f42480b\") " Nov 03 17:36:13.356737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.356710 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca" pod "5c982958-c024-4297-a336-c09c6f42480b" (UID: "5c982958-c024-4297-a336-c09c6f42480b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:13.358977 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.358951 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs" pod "5c982958-c024-4297-a336-c09c6f42480b" (UID: "5c982958-c024-4297-a336-c09c6f42480b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:13.395757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.395636 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2" pod "5c982958-c024-4297-a336-c09c6f42480b" (UID: "5c982958-c024-4297-a336-c09c6f42480b"). InnerVolumeSpecName "kube-api-access-g84f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:13.404522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.400555 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls" pod "5c982958-c024-4297-a336-c09c6f42480b" (UID: "5c982958-c024-4297-a336-c09c6f42480b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:13.421131 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.421086 4539 generic.go:356] "Generic (PLEG): container finished" podID="5c982958-c024-4297-a336-c09c6f42480b" containerID="a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701" exitCode=0 Nov 03 17:36:13.421328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.421155 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" Nov 03 17:36:13.421328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.421171 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" event={"ID":"5c982958-c024-4297-a336-c09c6f42480b","Type":"ContainerDied","Data":"a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701"} Nov 03 17:36:13.421328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.421221 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t" event={"ID":"5c982958-c024-4297-a336-c09c6f42480b","Type":"ContainerDied","Data":"d26956aed0f84ab4bcfb7b8b91369093418d514fb8c855f94f9c49980cd6539d"} Nov 03 17:36:13.421328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.421249 4539 scope.go:122] "RemoveContainer" containerID="a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701" Nov 03 17:36:13.439469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.438128 4539 scope.go:122] "RemoveContainer" containerID="a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701" Nov 03 17:36:13.439469 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:13.438613 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701\": container with ID starting with a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701 not found: ID does not exist" containerID="a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701" Nov 03 17:36:13.439469 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.438646 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701"} err="failed to get container status \"a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701\": rpc error: code = NotFound desc = could not find container \"a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701\": container with ID starting with a787a84e68ebc8e16ebfbbc71883028959d72979f2ad52cb42a26b9d5fb11701 not found: ID does not exist" Nov 03 17:36:13.457926 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.457891 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c982958-c024-4297-a336-c09c6f42480b-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:13.457926 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.457927 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:13.458185 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.457945 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84f2\" (UniqueName: \"kubernetes.io/projected/5c982958-c024-4297-a336-c09c6f42480b-kube-api-access-g84f2\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:13.458185 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.457958 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c982958-c024-4297-a336-c09c6f42480b-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:13.475910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.475878 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t"] Nov 03 17:36:13.480345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.480311 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1e9d48-6f9978bbd4-lvj9t"] Nov 03 17:36:13.793506 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:13.793474 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:14.069526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:14.069412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:14.468704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:14.468619 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c982958-c024-4297-a336-c09c6f42480b" path="/var/lib/kubelet/pods/5c982958-c024-4297-a336-c09c6f42480b/volumes" Nov 03 17:36:15.190654 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.190616 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" podUID="56322f04-5883-4048-bac9-338165b17c62" containerName="main" containerID="cri-o://513a403f7256170a513c21a2a04c6b65c44dfc42c509b335b9c6ff2c40394e61" gracePeriod=45 Nov 03 17:36:15.438226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.438194 4539 generic.go:356] "Generic (PLEG): container finished" podID="56322f04-5883-4048-bac9-338165b17c62" containerID="513a403f7256170a513c21a2a04c6b65c44dfc42c509b335b9c6ff2c40394e61" exitCode=0 Nov 03 17:36:15.438226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.438238 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" event={"ID":"56322f04-5883-4048-bac9-338165b17c62","Type":"ContainerDied","Data":"513a403f7256170a513c21a2a04c6b65c44dfc42c509b335b9c6ff2c40394e61"} Nov 03 17:36:15.452681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.452650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:15.470559 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.470504 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:36:15.581955 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.581903 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir\") pod \"56322f04-5883-4048-bac9-338165b17c62\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " Nov 03 17:36:15.581955 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.581965 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache\") pod \"56322f04-5883-4048-bac9-338165b17c62\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " Nov 03 17:36:15.582247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582089 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm\") pod \"56322f04-5883-4048-bac9-338165b17c62\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " Nov 03 17:36:15.582247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582165 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs\" (UniqueName: \"kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs\") pod \"56322f04-5883-4048-bac9-338165b17c62\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " Nov 03 17:36:15.582247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582205 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp\" (UniqueName: \"kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp\") pod \"56322f04-5883-4048-bac9-338165b17c62\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " Nov 03 17:36:15.582247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582238 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home\") pod \"56322f04-5883-4048-bac9-338165b17c62\" (UID: \"56322f04-5883-4048-bac9-338165b17c62\") " Nov 03 17:36:15.582482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582268 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache" pod "56322f04-5883-4048-bac9-338165b17c62" (UID: "56322f04-5883-4048-bac9-338165b17c62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:15.582482 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582283 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir" pod "56322f04-5883-4048-bac9-338165b17c62" (UID: "56322f04-5883-4048-bac9-338165b17c62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:15.582622 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582600 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home" pod "56322f04-5883-4048-bac9-338165b17c62" (UID: "56322f04-5883-4048-bac9-338165b17c62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:15.582739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582723 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:15.582798 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582745 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:15.582798 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.582758 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:15.585318 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.585293 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm" pod "56322f04-5883-4048-bac9-338165b17c62" (UID: "56322f04-5883-4048-bac9-338165b17c62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:15.585547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.585333 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs" pod "56322f04-5883-4048-bac9-338165b17c62" (UID: "56322f04-5883-4048-bac9-338165b17c62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:15.585547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.585368 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp" pod "56322f04-5883-4048-bac9-338165b17c62" (UID: "56322f04-5883-4048-bac9-338165b17c62"). InnerVolumeSpecName "kube-api-access-sfngp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:15.683354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.683316 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56322f04-5883-4048-bac9-338165b17c62-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:15.683354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.683349 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56322f04-5883-4048-bac9-338165b17c62-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:15.683354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:15.683359 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfngp\" (UniqueName: \"kubernetes.io/projected/56322f04-5883-4048-bac9-338165b17c62-kube-api-access-sfngp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:16.017487 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.017446 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:16.169726 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.169695 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:16.447847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.447808 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" event={"ID":"56322f04-5883-4048-bac9-338165b17c62","Type":"ContainerDied","Data":"4c2259c97e535ab5e5aa2b27c48a1781a306a0a9660410e88c6c6981320e289c"} Nov 03 17:36:16.447847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.447855 4539 scope.go:122] "RemoveContainer" containerID="513a403f7256170a513c21a2a04c6b65c44dfc42c509b335b9c6ff2c40394e61" Nov 03 17:36:16.448144 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.447816 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79" Nov 03 17:36:16.522241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.522205 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79"] Nov 03 17:36:16.537261 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.537229 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a9426b/sub-test-model-a9426b-kserve-84fd685b85-k6d79"] Nov 03 17:36:16.765100 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:16.765009 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:17.127379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.127344 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:17.388704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.388621 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:17.605980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.605941 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:17.790847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.790800 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:17.869178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.869134 4539 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:36:17.869380 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.869220 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:36:17.934568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:17.934530 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:18.267827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.267736 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" podUID="1fb3c920-1180-4861-aaa1-87df4bdc1516" containerName="main" containerID="cri-o://4ad95ca90a234a2b2f88b0c5bed748a3ee24f956c22fe012f98f0f35249c4188" gracePeriod=45 Nov 03 17:36:18.472198 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.472161 4539 generic.go:356] "Generic (PLEG): container finished" podID="1fb3c920-1180-4861-aaa1-87df4bdc1516" containerID="4ad95ca90a234a2b2f88b0c5bed748a3ee24f956c22fe012f98f0f35249c4188" exitCode=0 Nov 03 17:36:18.474385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.474353 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56322f04-5883-4048-bac9-338165b17c62" path="/var/lib/kubelet/pods/56322f04-5883-4048-bac9-338165b17c62/volumes" Nov 03 17:36:18.474728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.474707 4539 generic.go:356] "Generic (PLEG): container finished" podID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerID="96e6fd4363ceec7ff9187cacb19a72daead427afb21ac9da56964a6a01968db7" exitCode=0 Nov 03 17:36:18.474829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.474799 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" event={"ID":"1fb3c920-1180-4861-aaa1-87df4bdc1516","Type":"ContainerDied","Data":"4ad95ca90a234a2b2f88b0c5bed748a3ee24f956c22fe012f98f0f35249c4188"} Nov 03 17:36:18.474872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.474835 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" event={"ID":"93bc673a-89ea-43c1-85bd-57952e0c2848","Type":"ContainerDied","Data":"96e6fd4363ceec7ff9187cacb19a72daead427afb21ac9da56964a6a01968db7"} Nov 03 17:36:18.474923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.474873 4539 scope.go:122] "RemoveContainer" containerID="2253d66a488590b83b756961622ac959a6c3317b2e69fb73a64f242d2b69b23e" Nov 03 17:36:18.596825 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.596801 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:36:18.598866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.598848 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 17:36:18.715685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715641 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.715685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715690 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715747 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs\") pod \"1fb3c920-1180-4861-aaa1-87df4bdc1516\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715806 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715841 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm\") pod \"1fb3c920-1180-4861-aaa1-87df4bdc1516\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715870 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715956 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.715986 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir\") pod \"1fb3c920-1180-4861-aaa1-87df4bdc1516\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716039 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home\") pod \"1fb3c920-1180-4861-aaa1-87df4bdc1516\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716068 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716118 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache\") pod \"1fb3c920-1180-4861-aaa1-87df4bdc1516\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716146 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716177 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716213 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs\" (UniqueName: \"kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs\") pod \"1fb3c920-1180-4861-aaa1-87df4bdc1516\" (UID: \"1fb3c920-1180-4861-aaa1-87df4bdc1516\") " Nov 03 17:36:18.716241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716246 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token\") pod \"93bc673a-89ea-43c1-85bd-57952e0c2848\" (UID: \"93bc673a-89ea-43c1-85bd-57952e0c2848\") " Nov 03 17:36:18.716959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716375 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.716959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716700 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.716959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.716927 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.718128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.717514 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir" pod "1fb3c920-1180-4861-aaa1-87df4bdc1516" (UID: "1fb3c920-1180-4861-aaa1-87df4bdc1516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.718128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.717800 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:18.718128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.718054 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.718128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.718072 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home" pod "1fb3c920-1180-4861-aaa1-87df4bdc1516" (UID: "1fb3c920-1180-4861-aaa1-87df4bdc1516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.718128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.718105 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache" pod "1fb3c920-1180-4861-aaa1-87df4bdc1516" (UID: "1fb3c920-1180-4861-aaa1-87df4bdc1516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.718498 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.718147 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.720231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.720196 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:18.720731 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.720697 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs" pod "1fb3c920-1180-4861-aaa1-87df4bdc1516" (UID: "1fb3c920-1180-4861-aaa1-87df4bdc1516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:18.721021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.720951 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:36:18.721474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.721412 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.722536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.722491 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm" pod "1fb3c920-1180-4861-aaa1-87df4bdc1516" (UID: "1fb3c920-1180-4861-aaa1-87df4bdc1516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:18.722655 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.722629 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs" pod "1fb3c920-1180-4861-aaa1-87df4bdc1516" (UID: "1fb3c920-1180-4861-aaa1-87df4bdc1516"). InnerVolumeSpecName "kube-api-access-9vzgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:18.722768 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.722747 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf" pod "93bc673a-89ea-43c1-85bd-57952e0c2848" (UID: "93bc673a-89ea-43c1-85bd-57952e0c2848"). InnerVolumeSpecName "kube-api-access-rnvqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817767 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817797 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bc673a-89ea-43c1-85bd-57952e0c2848-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817808 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817817 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vzgs\" (UniqueName: \"kubernetes.io/projected/1fb3c920-1180-4861-aaa1-87df4bdc1516-kube-api-access-9vzgs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817826 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817836 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnvqf\" (UniqueName: \"kubernetes.io/projected/93bc673a-89ea-43c1-85bd-57952e0c2848-kube-api-access-rnvqf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817845 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817854 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb3c920-1180-4861-aaa1-87df4bdc1516-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817861 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817869 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.817865 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817877 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.818626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817885 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.818626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817892 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb3c920-1180-4861-aaa1-87df4bdc1516-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.818626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.817899 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bc673a-89ea-43c1-85bd-57952e0c2848-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:18.917866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.917829 4539 scope.go:122] "RemoveContainer" containerID="96e6fd4363ceec7ff9187cacb19a72daead427afb21ac9da56964a6a01968db7" Nov 03 17:36:18.929932 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.929908 4539 scope.go:122] "RemoveContainer" containerID="4ad95ca90a234a2b2f88b0c5bed748a3ee24f956c22fe012f98f0f35249c4188" Nov 03 17:36:18.978835 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:18.978801 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:19.481146 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.481104 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" event={"ID":"1fb3c920-1180-4861-aaa1-87df4bdc1516","Type":"ContainerDied","Data":"689ecdff2bb5fe6d3c3d53e2c54d9f89d634521d2094786fd57d1535bc95f0c4"} Nov 03 17:36:19.481309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.481165 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" event={"ID":"93bc673a-89ea-43c1-85bd-57952e0c2848","Type":"ContainerDied","Data":"24e0776a4034920640ec5fdffc2067824d0e8aafa322fb65fa109a0f4aea901b"} Nov 03 17:36:19.481309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.481194 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c" Nov 03 17:36:19.481309 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.481198 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk" Nov 03 17:36:19.610657 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.610568 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c"] Nov 03 17:36:19.619394 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.619356 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-ttg6c"] Nov 03 17:36:19.651007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.650972 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk"] Nov 03 17:36:19.653549 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:19.653524 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e2d835/sub-test-model-e2d835-kserve-c6b65d759-5g6hk"] Nov 03 17:36:20.467691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:20.467659 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb3c920-1180-4861-aaa1-87df4bdc1516" path="/var/lib/kubelet/pods/1fb3c920-1180-4861-aaa1-87df4bdc1516/volumes" Nov 03 17:36:20.468141 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:20.468127 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" path="/var/lib/kubelet/pods/93bc673a-89ea-43c1-85bd-57952e0c2848/volumes" Nov 03 17:36:20.636059 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:20.635980 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:20.786468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:20.786409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:21.578787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:21.578757 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:22.355055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:22.355020 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:22.572157 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:22.572127 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw"] Nov 03 17:36:22.725573 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:22.725497 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:23.431082 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:23.431018 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:23.821097 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:23.821055 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84"] Nov 03 17:36:23.821738 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:23.821301 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" podUID="b5311ef6-beae-4a0f-a463-951e90ff1f6f" containerName="maas-api" containerID="cri-o://8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883" gracePeriod=30 Nov 03 17:36:23.919345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:23.919310 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:24.121003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.120972 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:36:24.291723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.291683 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\") pod \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " Nov 03 17:36:24.291723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.291786 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca\") pod \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " Nov 03 17:36:24.291723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.291859 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd\" (UniqueName: \"kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd\") pod \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " Nov 03 17:36:24.291723 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.291900 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs\") pod \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\" (UID: \"b5311ef6-beae-4a0f-a463-951e90ff1f6f\") " Nov 03 17:36:24.292386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.292219 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca" pod "b5311ef6-beae-4a0f-a463-951e90ff1f6f" (UID: "b5311ef6-beae-4a0f-a463-951e90ff1f6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:24.292386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.292318 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5311ef6-beae-4a0f-a463-951e90ff1f6f-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:24.295297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.295248 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs" pod "b5311ef6-beae-4a0f-a463-951e90ff1f6f" (UID: "b5311ef6-beae-4a0f-a463-951e90ff1f6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:24.295676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.295651 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls" pod "b5311ef6-beae-4a0f-a463-951e90ff1f6f" (UID: "b5311ef6-beae-4a0f-a463-951e90ff1f6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:24.296600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.296570 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd" pod "b5311ef6-beae-4a0f-a463-951e90ff1f6f" (UID: "b5311ef6-beae-4a0f-a463-951e90ff1f6f"). InnerVolumeSpecName "kube-api-access-wp9nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:24.393099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.393018 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9nd\" (UniqueName: \"kubernetes.io/projected/b5311ef6-beae-4a0f-a463-951e90ff1f6f-kube-api-access-wp9nd\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:24.393099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.393045 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:24.393099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.393054 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5311ef6-beae-4a0f-a463-951e90ff1f6f-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:24.517639 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.517602 4539 generic.go:356] "Generic (PLEG): container finished" podID="b5311ef6-beae-4a0f-a463-951e90ff1f6f" containerID="8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883" exitCode=0 Nov 03 17:36:24.517797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.517683 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" Nov 03 17:36:24.517797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.517676 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" event={"ID":"b5311ef6-beae-4a0f-a463-951e90ff1f6f","Type":"ContainerDied","Data":"8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883"} Nov 03 17:36:24.517797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.517747 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84" event={"ID":"b5311ef6-beae-4a0f-a463-951e90ff1f6f","Type":"ContainerDied","Data":"1326a25033bd6ee4e44f016a3487fa058300b59ed145731ccc488d47dc427a2e"} Nov 03 17:36:24.517797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.517773 4539 scope.go:122] "RemoveContainer" containerID="8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883" Nov 03 17:36:24.538623 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.538600 4539 scope.go:122] "RemoveContainer" containerID="8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883" Nov 03 17:36:24.538926 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:24.538899 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883\": container with ID starting with 8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883 not found: ID does not exist" containerID="8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883" Nov 03 17:36:24.539116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.539082 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883"} err="failed to get container status \"8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883\": rpc error: code = NotFound desc = could not find container \"8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883\": container with ID starting with 8b0812592cdfc67cd455011ce5eebcf7fa1a9933dde7ea332d17d28e274f2883 not found: ID does not exist" Nov 03 17:36:24.576951 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.576921 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84"] Nov 03 17:36:24.580471 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.580448 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a0e27-77d87865f8-nqc84"] Nov 03 17:36:24.671743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:24.671607 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:25.589924 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:25.589890 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:25.748165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:25.748121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:26.157960 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:26.157923 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:36:26.272284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:26.272246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:26.470739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:26.470664 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5311ef6-beae-4a0f-a463-951e90ff1f6f" path="/var/lib/kubelet/pods/b5311ef6-beae-4a0f-a463-951e90ff1f6f/volumes" Nov 03 17:36:27.575116 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:27.575077 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts"] Nov 03 17:36:27.575628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:27.575361 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" podUID="708707d8-acb2-4982-9c2c-52e48776d590" containerName="maas-api" containerID="cri-o://238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea" gracePeriod=30 Nov 03 17:36:27.746496 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:27.746461 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:27.882365 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:27.882341 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 17:36:27.964541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:27.964510 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:28.033332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.033299 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls\") pod \"708707d8-acb2-4982-9c2c-52e48776d590\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " Nov 03 17:36:28.033531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.033484 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs\") pod \"708707d8-acb2-4982-9c2c-52e48776d590\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " Nov 03 17:36:28.033580 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.033552 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz\" (UniqueName: \"kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz\") pod \"708707d8-acb2-4982-9c2c-52e48776d590\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " Nov 03 17:36:28.033626 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.033611 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca\") pod \"708707d8-acb2-4982-9c2c-52e48776d590\" (UID: \"708707d8-acb2-4982-9c2c-52e48776d590\") " Nov 03 17:36:28.034092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.034060 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca" pod "708707d8-acb2-4982-9c2c-52e48776d590" (UID: "708707d8-acb2-4982-9c2c-52e48776d590"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:28.035707 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.035680 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls" pod "708707d8-acb2-4982-9c2c-52e48776d590" (UID: "708707d8-acb2-4982-9c2c-52e48776d590"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:28.035931 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.035900 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs" pod "708707d8-acb2-4982-9c2c-52e48776d590" (UID: "708707d8-acb2-4982-9c2c-52e48776d590"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:28.036170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.036149 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz" pod "708707d8-acb2-4982-9c2c-52e48776d590" (UID: "708707d8-acb2-4982-9c2c-52e48776d590"). InnerVolumeSpecName "kube-api-access-mn9wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:28.135393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.135310 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.135393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.135338 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn9wz\" (UniqueName: \"kubernetes.io/projected/708707d8-acb2-4982-9c2c-52e48776d590-kube-api-access-mn9wz\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.135393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.135348 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708707d8-acb2-4982-9c2c-52e48776d590-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.135393 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.135357 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708707d8-acb2-4982-9c2c-52e48776d590-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.361396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.361358 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:28.544854 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.544825 4539 generic.go:356] "Generic (PLEG): container finished" podID="708707d8-acb2-4982-9c2c-52e48776d590" containerID="238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea" exitCode=0 Nov 03 17:36:28.545381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.544882 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" Nov 03 17:36:28.545381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.544912 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" event={"ID":"708707d8-acb2-4982-9c2c-52e48776d590","Type":"ContainerDied","Data":"238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea"} Nov 03 17:36:28.545381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.544967 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts" event={"ID":"708707d8-acb2-4982-9c2c-52e48776d590","Type":"ContainerDied","Data":"b51abe98039f85815c978240fce05572d7c9df5aba6330c51ec099bde76ffd41"} Nov 03 17:36:28.545381 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.544990 4539 scope.go:122] "RemoveContainer" containerID="238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea" Nov 03 17:36:28.556554 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.556537 4539 scope.go:122] "RemoveContainer" containerID="238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea" Nov 03 17:36:28.556864 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:28.556837 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea\": container with ID starting with 238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea not found: ID does not exist" containerID="238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea" Nov 03 17:36:28.556972 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.556867 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea"} err="failed to get container status \"238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea\": rpc error: code = NotFound desc = could not find container \"238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea\": container with ID starting with 238363c62a1f49b927c9a6786281ecc15013a06f2826e5615f00f9a451f1f6ea not found: ID does not exist" Nov 03 17:36:28.597690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.597654 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts"] Nov 03 17:36:28.610640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.610610 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c6f303-6c856b455b-zkhts"] Nov 03 17:36:28.707172 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.707134 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:28.861967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:28.861927 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:30.195704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:30.195671 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:30.467411 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:30.467332 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708707d8-acb2-4982-9c2c-52e48776d590" path="/var/lib/kubelet/pods/708707d8-acb2-4982-9c2c-52e48776d590/volumes" Nov 03 17:36:30.532730 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:30.532679 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:31.845121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:31.845081 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:33.463775 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:33.463741 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:34.702630 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:34.702594 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:34.854400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:34.854367 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:35.081016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:35.080988 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:35.417534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:35.417413 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:35.569413 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:35.569373 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:35.826656 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:35.826626 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:36.234863 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:36.234767 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:36.583762 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:36.583727 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:36.747310 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:36.747274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:36.956165 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:36.956075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:37.125965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.125926 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:37.288839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.288803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:37.593531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.593438 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" containerID="cri-o://b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d" gracePeriod=45 Nov 03 17:36:37.853797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.853735 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 17:36:37.932787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.932617 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir\") pod \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " Nov 03 17:36:37.933344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933111 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm\") pod \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " Nov 03 17:36:37.933344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933246 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b\" (UniqueName: \"kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b\") pod \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " Nov 03 17:36:37.933344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933287 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home\") pod \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " Nov 03 17:36:37.933665 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933522 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir" pod "c2469139-15c6-45eb-ad72-ab7f4ef38e4e" (UID: "c2469139-15c6-45eb-ad72-ab7f4ef38e4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933768 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache\") pod \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933792 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home" pod "c2469139-15c6-45eb-ad72-ab7f4ef38e4e" (UID: "c2469139-15c6-45eb-ad72-ab7f4ef38e4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.933857 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs\") pod \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\" (UID: \"c2469139-15c6-45eb-ad72-ab7f4ef38e4e\") " Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.934018 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache" pod "c2469139-15c6-45eb-ad72-ab7f4ef38e4e" (UID: "c2469139-15c6-45eb-ad72-ab7f4ef38e4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.934232 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.934248 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:37.934255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.934261 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:37.936266 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.936239 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm" pod "c2469139-15c6-45eb-ad72-ab7f4ef38e4e" (UID: "c2469139-15c6-45eb-ad72-ab7f4ef38e4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:37.936561 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.936524 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b" pod "c2469139-15c6-45eb-ad72-ab7f4ef38e4e" (UID: "c2469139-15c6-45eb-ad72-ab7f4ef38e4e"). InnerVolumeSpecName "kube-api-access-zcj7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:37.937065 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:37.937030 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs" pod "c2469139-15c6-45eb-ad72-ab7f4ef38e4e" (UID: "c2469139-15c6-45eb-ad72-ab7f4ef38e4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:38.035869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.035828 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.035869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.035867 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.036464 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.035882 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcj7b\" (UniqueName: \"kubernetes.io/projected/c2469139-15c6-45eb-ad72-ab7f4ef38e4e-kube-api-access-zcj7b\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.605843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.605803 4539 generic.go:356] "Generic (PLEG): container finished" podID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerID="b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d" exitCode=0 Nov 03 17:36:38.606024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.605853 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" Nov 03 17:36:38.606024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.605889 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" event={"ID":"c2469139-15c6-45eb-ad72-ab7f4ef38e4e","Type":"ContainerDied","Data":"b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d"} Nov 03 17:36:38.606024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.605932 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw" event={"ID":"c2469139-15c6-45eb-ad72-ab7f4ef38e4e","Type":"ContainerDied","Data":"4fee29f13d23b8184727ea3982f534d80c2d8f0a5b6831d91ff3786e06e726d5"} Nov 03 17:36:38.606024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.605948 4539 scope.go:122] "RemoveContainer" containerID="b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d" Nov 03 17:36:38.615434 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.615401 4539 scope.go:122] "RemoveContainer" containerID="b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d" Nov 03 17:36:38.615728 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:38.615705 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d\": container with ID starting with b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d not found: ID does not exist" containerID="b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d" Nov 03 17:36:38.615817 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.615735 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d"} err="failed to get container status \"b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d\": rpc error: code = NotFound desc = could not find container \"b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d\": container with ID starting with b6e72297381772a781272d6cc864b2e7c6b1e8ca439ce0d244dcfcd2e7014b4d not found: ID does not exist" Nov 03 17:36:38.657483 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.657449 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw"] Nov 03 17:36:38.669563 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.669544 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/premium-w0-71d7b4-kserve-5995b86d95-gqmkw"] Nov 03 17:36:38.681880 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:38.681838 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:40.192325 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:40.192252 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:40.467818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:40.467737 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" path="/var/lib/kubelet/pods/c2469139-15c6-45eb-ad72-ab7f4ef38e4e/volumes" Nov 03 17:36:41.785038 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:41.785003 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:42.065844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:42.065751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:42.072373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:42.072340 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx"] Nov 03 17:36:42.102229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:42.102197 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs"] Nov 03 17:36:42.220025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:42.219953 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:42.368133 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:42.368050 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:42.884366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:42.884332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:43.033961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:43.033912 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:43.549339 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:43.549296 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:44.101024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:44.100994 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:44.809994 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:44.809955 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98"] Nov 03 17:36:44.810465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:44.810156 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" podUID="ccb45b49-1a1d-4825-beb5-4cf8f983ce97" containerName="maas-api" containerID="cri-o://d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142" gracePeriod=30 Nov 03 17:36:44.927739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:44.927707 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:45.109526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.109503 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 17:36:45.184547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.184496 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp"] Nov 03 17:36:45.184844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.184811 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" podUID="c711f644-5999-424c-9af3-2f43cb718841" containerName="maas-api" containerID="cri-o://2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559" gracePeriod=30 Nov 03 17:36:45.221078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.221048 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\") pod \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " Nov 03 17:36:45.221236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.221116 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca\") pod \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " Nov 03 17:36:45.221236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.221171 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8\" (UniqueName: \"kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8\") pod \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " Nov 03 17:36:45.221236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.221217 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs\") pod \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\" (UID: \"ccb45b49-1a1d-4825-beb5-4cf8f983ce97\") " Nov 03 17:36:45.222336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.222285 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca" pod "ccb45b49-1a1d-4825-beb5-4cf8f983ce97" (UID: "ccb45b49-1a1d-4825-beb5-4cf8f983ce97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:45.224502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.224462 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs" pod "ccb45b49-1a1d-4825-beb5-4cf8f983ce97" (UID: "ccb45b49-1a1d-4825-beb5-4cf8f983ce97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:45.224725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.224699 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls" pod "ccb45b49-1a1d-4825-beb5-4cf8f983ce97" (UID: "ccb45b49-1a1d-4825-beb5-4cf8f983ce97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:45.225292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.224993 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8" pod "ccb45b49-1a1d-4825-beb5-4cf8f983ce97" (UID: "ccb45b49-1a1d-4825-beb5-4cf8f983ce97"). InnerVolumeSpecName "kube-api-access-b9ws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:45.283677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.283645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:45.322601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.322567 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.322601 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.322602 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ws8\" (UniqueName: \"kubernetes.io/projected/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-kube-api-access-b9ws8\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.322805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.322619 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.322805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.322632 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb45b49-1a1d-4825-beb5-4cf8f983ce97-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.475574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.475545 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:36:45.525130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.525099 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs\") pod \"c711f644-5999-424c-9af3-2f43cb718841\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " Nov 03 17:36:45.525300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.525151 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca\") pod \"c711f644-5999-424c-9af3-2f43cb718841\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " Nov 03 17:36:45.525300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.525292 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\") pod \"c711f644-5999-424c-9af3-2f43cb718841\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " Nov 03 17:36:45.525392 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.525333 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp\" (UniqueName: \"kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp\") pod \"c711f644-5999-424c-9af3-2f43cb718841\" (UID: \"c711f644-5999-424c-9af3-2f43cb718841\") " Nov 03 17:36:45.525639 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.525608 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca" pod "c711f644-5999-424c-9af3-2f43cb718841" (UID: "c711f644-5999-424c-9af3-2f43cb718841"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:45.525807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.525764 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c711f644-5999-424c-9af3-2f43cb718841-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.527721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.527694 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs" pod "c711f644-5999-424c-9af3-2f43cb718841" (UID: "c711f644-5999-424c-9af3-2f43cb718841"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:45.528180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.528049 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls" pod "c711f644-5999-424c-9af3-2f43cb718841" (UID: "c711f644-5999-424c-9af3-2f43cb718841"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:45.528180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.528059 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp" pod "c711f644-5999-424c-9af3-2f43cb718841" (UID: "c711f644-5999-424c-9af3-2f43cb718841"). InnerVolumeSpecName "kube-api-access-zfvkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:45.627031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.626937 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.627031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.626965 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c711f644-5999-424c-9af3-2f43cb718841-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.627031 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.626974 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvkp\" (UniqueName: \"kubernetes.io/projected/c711f644-5999-424c-9af3-2f43cb718841-kube-api-access-zfvkp\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.674864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.674815 4539 generic.go:356] "Generic (PLEG): container finished" podID="ccb45b49-1a1d-4825-beb5-4cf8f983ce97" containerID="d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142" exitCode=0 Nov 03 17:36:45.675075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.674867 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" Nov 03 17:36:45.675075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.674899 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" event={"ID":"ccb45b49-1a1d-4825-beb5-4cf8f983ce97","Type":"ContainerDied","Data":"d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142"} Nov 03 17:36:45.675075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.674958 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98" event={"ID":"ccb45b49-1a1d-4825-beb5-4cf8f983ce97","Type":"ContainerDied","Data":"2a56463e45c77c051e80d5c437da236cad398017d2974f6c863a6d1eb9996cc4"} Nov 03 17:36:45.675075 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.674985 4539 scope.go:122] "RemoveContainer" containerID="d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142" Nov 03 17:36:45.677544 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.677495 4539 generic.go:356] "Generic (PLEG): container finished" podID="c711f644-5999-424c-9af3-2f43cb718841" containerID="2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559" exitCode=0 Nov 03 17:36:45.677681 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.677651 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" event={"ID":"c711f644-5999-424c-9af3-2f43cb718841","Type":"ContainerDied","Data":"2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559"} Nov 03 17:36:45.677751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.677697 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" event={"ID":"c711f644-5999-424c-9af3-2f43cb718841","Type":"ContainerDied","Data":"9eaf04e0b0798404a62329d2ec2edaf0fbb6c18c0525810bef78921e6bfc5280"} Nov 03 17:36:45.677818 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.677769 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp" Nov 03 17:36:45.687866 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.687840 4539 scope.go:122] "RemoveContainer" containerID="d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142" Nov 03 17:36:45.688205 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:45.688185 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142\": container with ID starting with d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142 not found: ID does not exist" containerID="d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142" Nov 03 17:36:45.688297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.688215 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142"} err="failed to get container status \"d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142\": rpc error: code = NotFound desc = could not find container \"d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142\": container with ID starting with d1801591c2d548fb670c09144490e53aa96f8c9995174734ee0700ada795a142 not found: ID does not exist" Nov 03 17:36:45.688297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.688233 4539 scope.go:122] "RemoveContainer" containerID="2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559" Nov 03 17:36:45.698041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.698023 4539 scope.go:122] "RemoveContainer" containerID="2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559" Nov 03 17:36:45.698327 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:45.698309 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559\": container with ID starting with 2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559 not found: ID does not exist" containerID="2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559" Nov 03 17:36:45.698379 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.698335 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559"} err="failed to get container status \"2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559\": rpc error: code = NotFound desc = could not find container \"2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559\": container with ID starting with 2d53f8be108234a1a8c1ebc836c4715cf30d27a5c36a9f190b9cb204284ab559 not found: ID does not exist" Nov 03 17:36:45.727364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.727337 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98"] Nov 03 17:36:45.731864 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.731843 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-71d7b4-74867c45d4-wzg98"] Nov 03 17:36:45.776685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.776656 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp"] Nov 03 17:36:45.784950 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.784928 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-790111-56645dfc6d-xd8lp"] Nov 03 17:36:45.923348 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:45.923238 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:46.468020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:46.467989 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c711f644-5999-424c-9af3-2f43cb718841" path="/var/lib/kubelet/pods/c711f644-5999-424c-9af3-2f43cb718841/volumes" Nov 03 17:36:46.468409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:46.468388 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb45b49-1a1d-4825-beb5-4cf8f983ce97" path="/var/lib/kubelet/pods/ccb45b49-1a1d-4825-beb5-4cf8f983ce97/volumes" Nov 03 17:36:47.602326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:47.602290 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:49.375751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:49.375718 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:51.232462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:51.232408 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:51.415785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:51.415743 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:52.096639 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:52.096601 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:52.881934 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:52.881880 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:54.555004 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:54.554977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:56.147800 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:56.147762 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:56.157455 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:56.157385 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:36:56.445475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:56.445344 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:57.097080 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.097025 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" containerID="cri-o://27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e" gracePeriod=45 Nov 03 17:36:57.127946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.127877 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" containerID="cri-o://443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba" gracePeriod=45 Nov 03 17:36:57.404465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.404418 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 17:36:57.437366 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.437339 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 17:36:57.553500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553466 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir\") pod \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " Nov 03 17:36:57.553700 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553523 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs\") pod \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " Nov 03 17:36:57.553700 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553606 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home\") pod \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " Nov 03 17:36:57.553700 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553661 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache\") pod \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " Nov 03 17:36:57.553700 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553694 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home\") pod \"de7edd38-07cf-4455-ba6d-f2c502452d37\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553734 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4\" (UniqueName: \"kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4\") pod \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553775 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir\") pod \"de7edd38-07cf-4455-ba6d-f2c502452d37\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553815 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir" pod "0e552f18-887c-49cf-88c4-f043aa0d5ddf" (UID: "0e552f18-887c-49cf-88c4-f043aa0d5ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553842 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9\" (UniqueName: \"kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9\") pod \"de7edd38-07cf-4455-ba6d-f2c502452d37\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553886 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache\") pod \"de7edd38-07cf-4455-ba6d-f2c502452d37\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553916 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm\") pod \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\" (UID: \"0e552f18-887c-49cf-88c4-f043aa0d5ddf\") " Nov 03 17:36:57.553959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553927 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home" pod "0e552f18-887c-49cf-88c4-f043aa0d5ddf" (UID: "0e552f18-887c-49cf-88c4-f043aa0d5ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.554328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.553979 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs\" (UniqueName: \"kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs\") pod \"de7edd38-07cf-4455-ba6d-f2c502452d37\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " Nov 03 17:36:57.554328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.554017 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm\") pod \"de7edd38-07cf-4455-ba6d-f2c502452d37\" (UID: \"de7edd38-07cf-4455-ba6d-f2c502452d37\") " Nov 03 17:36:57.554328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.554158 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache" pod "de7edd38-07cf-4455-ba6d-f2c502452d37" (UID: "de7edd38-07cf-4455-ba6d-f2c502452d37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.554328 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.554290 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir" pod "de7edd38-07cf-4455-ba6d-f2c502452d37" (UID: "de7edd38-07cf-4455-ba6d-f2c502452d37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.554652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.554626 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache" pod "0e552f18-887c-49cf-88c4-f043aa0d5ddf" (UID: "0e552f18-887c-49cf-88c4-f043aa0d5ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.555265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.554701 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home" pod "de7edd38-07cf-4455-ba6d-f2c502452d37" (UID: "de7edd38-07cf-4455-ba6d-f2c502452d37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.555265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.555130 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.555265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.555154 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.555265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.555166 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.555265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.555179 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.556910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.556889 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9" pod "de7edd38-07cf-4455-ba6d-f2c502452d37" (UID: "de7edd38-07cf-4455-ba6d-f2c502452d37"). InnerVolumeSpecName "kube-api-access-g5gf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:57.556910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.556886 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm" pod "de7edd38-07cf-4455-ba6d-f2c502452d37" (UID: "de7edd38-07cf-4455-ba6d-f2c502452d37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.557230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.557209 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs" pod "de7edd38-07cf-4455-ba6d-f2c502452d37" (UID: "de7edd38-07cf-4455-ba6d-f2c502452d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:57.557548 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.557525 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm" pod "0e552f18-887c-49cf-88c4-f043aa0d5ddf" (UID: "0e552f18-887c-49cf-88c4-f043aa0d5ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.557795 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.557774 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs" pod "0e552f18-887c-49cf-88c4-f043aa0d5ddf" (UID: "0e552f18-887c-49cf-88c4-f043aa0d5ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:57.559321 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.559298 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4" pod "0e552f18-887c-49cf-88c4-f043aa0d5ddf" (UID: "0e552f18-887c-49cf-88c4-f043aa0d5ddf"). InnerVolumeSpecName "kube-api-access-9sht4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655869 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655899 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655909 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sht4\" (UniqueName: \"kubernetes.io/projected/0e552f18-887c-49cf-88c4-f043aa0d5ddf-kube-api-access-9sht4\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655921 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gf9\" (UniqueName: \"kubernetes.io/projected/de7edd38-07cf-4455-ba6d-f2c502452d37-kube-api-access-g5gf9\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655929 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e552f18-887c-49cf-88c4-f043aa0d5ddf-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655937 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7edd38-07cf-4455-ba6d-f2c502452d37-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655945 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7edd38-07cf-4455-ba6d-f2c502452d37-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.655949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.655954 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e552f18-887c-49cf-88c4-f043aa0d5ddf-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.751322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.751287 4539 generic.go:356] "Generic (PLEG): container finished" podID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerID="443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba" exitCode=0 Nov 03 17:36:57.751531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.751341 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" Nov 03 17:36:57.751531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.751361 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" event={"ID":"0e552f18-887c-49cf-88c4-f043aa0d5ddf","Type":"ContainerDied","Data":"443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba"} Nov 03 17:36:57.751531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.751401 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs" event={"ID":"0e552f18-887c-49cf-88c4-f043aa0d5ddf","Type":"ContainerDied","Data":"d03df31a1c3b144158ede92e55e4d4269046c60c679dd9f17f72b61407fcde70"} Nov 03 17:36:57.751531 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.751434 4539 scope.go:122] "RemoveContainer" containerID="443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba" Nov 03 17:36:57.752802 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.752770 4539 generic.go:356] "Generic (PLEG): container finished" podID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerID="27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e" exitCode=0 Nov 03 17:36:57.752911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.752826 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" Nov 03 17:36:57.752911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.752841 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" event={"ID":"de7edd38-07cf-4455-ba6d-f2c502452d37","Type":"ContainerDied","Data":"27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e"} Nov 03 17:36:57.752911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.752872 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" event={"ID":"de7edd38-07cf-4455-ba6d-f2c502452d37","Type":"ContainerDied","Data":"f99e4db4daa046ef4ea3f517c4f6c0dbdd2617b0cbb019c539497df1229ddd3d"} Nov 03 17:36:57.761143 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.761126 4539 scope.go:122] "RemoveContainer" containerID="443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba" Nov 03 17:36:57.761392 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:57.761373 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba\": container with ID starting with 443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba not found: ID does not exist" containerID="443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba" Nov 03 17:36:57.761505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.761402 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba"} err="failed to get container status \"443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba\": rpc error: code = NotFound desc = could not find container \"443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba\": container with ID starting with 443b926e46b721075b9d06268a2a917ac46d1dfeb58a9ec94f2bf79973ec11ba not found: ID does not exist" Nov 03 17:36:57.761505 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.761437 4539 scope.go:122] "RemoveContainer" containerID="27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e" Nov 03 17:36:57.771230 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.771213 4539 scope.go:122] "RemoveContainer" containerID="27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e" Nov 03 17:36:57.771532 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:36:57.771515 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e\": container with ID starting with 27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e not found: ID does not exist" containerID="27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e" Nov 03 17:36:57.771600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.771541 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e"} err="failed to get container status \"27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e\": rpc error: code = NotFound desc = could not find container \"27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e\": container with ID starting with 27c70610692892654fd0cb244f32890ab12d0cc403b949258f90ade25e3f948e not found: ID does not exist" Nov 03 17:36:57.805688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.805661 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx"] Nov 03 17:36:57.810109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.810083 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx"] Nov 03 17:36:57.844645 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.844617 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs"] Nov 03 17:36:57.848813 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.848785 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-460104/simulator-w4-460104-kserve-5756956476-gsdgs"] Nov 03 17:36:57.892014 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:57.891974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:36:58.372859 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:58.372772 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-2-w4-460104-kserve-7557d5d887-5n8vx" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:36:58.468608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:58.468575 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" path="/var/lib/kubelet/pods/0e552f18-887c-49cf-88c4-f043aa0d5ddf/volumes" Nov 03 17:36:58.468997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:58.468983 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" path="/var/lib/kubelet/pods/de7edd38-07cf-4455-ba6d-f2c502452d37/volumes" Nov 03 17:36:59.583222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:36:59.583189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:00.106060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:00.106028 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:00.249291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:00.249259 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:00.523076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:00.522399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:00.670947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:00.670910 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:00.923564 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:00.923379 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:01.107528 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:01.107476 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:01.265327 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:01.265228 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:01.466231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:01.466197 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:02.232184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:02.232138 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:02.953108 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:02.953057 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:03.661576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:03.661544 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:03.923488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:03.923369 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:05.035353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:05.035301 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:05.695112 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:05.695063 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:07.337228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:07.337191 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:07.478551 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:07.478518 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:07.634239 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:07.634153 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:08.036084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:08.036048 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:08.385944 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:08.385857 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:08.975037 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:08.975005 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:09.137091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:09.137060 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:09.284185 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:09.284152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:09.486186 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:09.486151 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:11.048931 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:11.048890 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:11.244146 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:11.244118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:12.862184 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:12.862149 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:14.394712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:14.394671 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c"] Nov 03 17:37:14.395332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:14.395135 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:14.435297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:14.435262 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8"] Nov 03 17:37:14.611882 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:14.611843 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:14.917143 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:14.917114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:15.889024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:15.888995 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn"] Nov 03 17:37:15.889615 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:15.889232 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" podUID="fc59b182-642d-4931-8dad-2a1885caf806" containerName="maas-api" containerID="cri-o://1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8" gracePeriod=30 Nov 03 17:37:15.995728 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:15.995694 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:16.179040 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.179014 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:37:16.249125 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.249088 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs\") pod \"fc59b182-642d-4931-8dad-2a1885caf806\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " Nov 03 17:37:16.249291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.249171 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls\") pod \"fc59b182-642d-4931-8dad-2a1885caf806\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " Nov 03 17:37:16.249291 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.249258 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca\") pod \"fc59b182-642d-4931-8dad-2a1885caf806\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " Nov 03 17:37:16.249384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.249340 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt\" (UniqueName: \"kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt\") pod \"fc59b182-642d-4931-8dad-2a1885caf806\" (UID: \"fc59b182-642d-4931-8dad-2a1885caf806\") " Nov 03 17:37:16.249685 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.249654 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca" pod "fc59b182-642d-4931-8dad-2a1885caf806" (UID: "fc59b182-642d-4931-8dad-2a1885caf806"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:16.249829 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.249769 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc59b182-642d-4931-8dad-2a1885caf806-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:16.251583 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.251552 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs" pod "fc59b182-642d-4931-8dad-2a1885caf806" (UID: "fc59b182-642d-4931-8dad-2a1885caf806"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:16.251687 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.251655 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt" pod "fc59b182-642d-4931-8dad-2a1885caf806" (UID: "fc59b182-642d-4931-8dad-2a1885caf806"). InnerVolumeSpecName "kube-api-access-v4wpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:16.251877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.251858 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls" pod "fc59b182-642d-4931-8dad-2a1885caf806" (UID: "fc59b182-642d-4931-8dad-2a1885caf806"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:16.351003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.350884 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4wpt\" (UniqueName: \"kubernetes.io/projected/fc59b182-642d-4931-8dad-2a1885caf806-kube-api-access-v4wpt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:16.351003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.350962 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:16.351003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.350977 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc59b182-642d-4931-8dad-2a1885caf806-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:16.596418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.596379 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:16.782673 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.782632 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:16.867561 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.867528 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" event={"ID":"fc59b182-642d-4931-8dad-2a1885caf806","Type":"ContainerDied","Data":"1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8"} Nov 03 17:37:16.867920 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.867900 4539 scope.go:122] "RemoveContainer" containerID="1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8" Nov 03 17:37:16.868218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.868202 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" Nov 03 17:37:16.868453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.867361 4539 generic.go:356] "Generic (PLEG): container finished" podID="fc59b182-642d-4931-8dad-2a1885caf806" containerID="1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8" exitCode=0 Nov 03 17:37:16.868733 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.868624 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn" event={"ID":"fc59b182-642d-4931-8dad-2a1885caf806","Type":"ContainerDied","Data":"8f74c7fa3ebe3b834d1ee1de0703fffca48f84e59341a33c09d323e2762bbb9a"} Nov 03 17:37:16.882475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.882446 4539 scope.go:122] "RemoveContainer" containerID="1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8" Nov 03 17:37:16.882828 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:37:16.882806 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8\": container with ID starting with 1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8 not found: ID does not exist" containerID="1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8" Nov 03 17:37:16.882912 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.882838 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8"} err="failed to get container status \"1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8\": rpc error: code = NotFound desc = could not find container \"1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8\": container with ID starting with 1000f80eef61047a55d3ff6ad17e377e709e50810a995463b32085a7748342b8 not found: ID does not exist" Nov 03 17:37:16.924741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.924705 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn"] Nov 03 17:37:16.936454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:16.936407 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e2d835-dbb89645c-c8lqn"] Nov 03 17:37:17.998572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:17.998540 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:18.275500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:18.275470 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:18.467767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:18.467732 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc59b182-642d-4931-8dad-2a1885caf806" path="/var/lib/kubelet/pods/fc59b182-642d-4931-8dad-2a1885caf806/volumes" Nov 03 17:37:19.395969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:19.395924 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerName="payload-processing" containerID="cri-o://784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360" gracePeriod=25 Nov 03 17:37:19.435709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:19.435660 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerName="payload-pre-processing" containerID="cri-o://a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed" gracePeriod=25 Nov 03 17:37:19.893442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:19.893389 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:20.033858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:20.033816 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:20.268281 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:20.268187 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5"] Nov 03 17:37:20.268500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:20.268417 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" containerID="cri-o://9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88" gracePeriod=30 Nov 03 17:37:20.292409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:20.292375 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:20.292881 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:20.292852 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:21.296333 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:21.296289 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:21.735769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:21.735678 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:22.518477 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:22.518401 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:22.902691 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:22.902657 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:23.348641 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:23.348590 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:23.763410 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:23.763332 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:23.914986 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:23.914948 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:24.393975 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:24.393937 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:24.696539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:24.696463 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw"] Nov 03 17:37:24.696688 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:24.696666 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" containerID="cri-o://a83f8694195e00bce48b9b51e3841cecad5413f5d90545951477e70be62b6795" gracePeriod=30 Nov 03 17:37:24.757849 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:24.757817 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:25.178599 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.178557 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:25.347048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.347014 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:25.536616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.536591 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:37:25.641790 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641753 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641800 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641831 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641866 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641898 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641923 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.641999 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642029 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642051 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket\") pod \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\" (UID: \"5929f511-5e10-44ff-94d3-68c7d7a6e6cb\") " Nov 03 17:37:25.642418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642121 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.642418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642206 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.642418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642368 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.642418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642409 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.642729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642447 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.642729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642453 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:25.643063 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.642468 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.645311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.645272 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:25.646055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.646018 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "kube-api-access-nts5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:25.646808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.646778 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.647035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.647017 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token" pod "5929f511-5e10-44ff-94d3-68c7d7a6e6cb" (UID: "5929f511-5e10-44ff-94d3-68c7d7a6e6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743831 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743870 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nts5t\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-kube-api-access-nts5t\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743881 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743889 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743897 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743905 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.743922 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.743913 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5929f511-5e10-44ff-94d3-68c7d7a6e6cb-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.922760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.922728 4539 generic.go:356] "Generic (PLEG): container finished" podID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerID="9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88" exitCode=0 Nov 03 17:37:25.922967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.922773 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" Nov 03 17:37:25.922967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.922813 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" event={"ID":"5929f511-5e10-44ff-94d3-68c7d7a6e6cb","Type":"ContainerDied","Data":"9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88"} Nov 03 17:37:25.922967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.922880 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5" event={"ID":"5929f511-5e10-44ff-94d3-68c7d7a6e6cb","Type":"ContainerDied","Data":"f69ffa1a02cb98b7e367906e6a8e5fff5b056d696f17647599d076315b64a411"} Nov 03 17:37:25.922967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.922913 4539 scope.go:122] "RemoveContainer" containerID="9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88" Nov 03 17:37:25.932947 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.932926 4539 scope.go:122] "RemoveContainer" containerID="9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88" Nov 03 17:37:25.933255 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:37:25.933234 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88\": container with ID starting with 9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88 not found: ID does not exist" containerID="9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88" Nov 03 17:37:25.933353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.933271 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88"} err="failed to get container status \"9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88\": rpc error: code = NotFound desc = could not find container \"9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88\": container with ID starting with 9f68a97a751c09b466378d3f2b302c242dae06cd3a7a51fdf03d509a72658b88 not found: ID does not exist" Nov 03 17:37:25.974842 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.974809 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5"] Nov 03 17:37:25.979593 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:25.979568 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wlxs5"] Nov 03 17:37:26.156180 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:37:26.156144 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:37:26.467146 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:26.467060 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" path="/var/lib/kubelet/pods/5929f511-5e10-44ff-94d3-68c7d7a6e6cb/volumes" Nov 03 17:37:26.882991 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:26.882958 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:27.633287 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:27.633252 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:27.805720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:27.805668 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:28.476300 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:28.476219 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:29.950665 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:29.950627 4539 generic.go:356] "Generic (PLEG): container finished" podID="feacadd3-6ac7-4550-9b6b-a786de858052" containerID="a83f8694195e00bce48b9b51e3841cecad5413f5d90545951477e70be62b6795" exitCode=0 Nov 03 17:37:29.951092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:29.950839 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" event={"ID":"feacadd3-6ac7-4550-9b6b-a786de858052","Type":"ContainerDied","Data":"a83f8694195e00bce48b9b51e3841cecad5413f5d90545951477e70be62b6795"} Nov 03 17:37:29.951092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:29.950942 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" event={"ID":"feacadd3-6ac7-4550-9b6b-a786de858052","Type":"ContainerDied","Data":"81e1b75c0bf57135cba29a683e124b22f99da0c55b1cc29ee963be508eb13a72"} Nov 03 17:37:29.951092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:29.950977 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e1b75c0bf57135cba29a683e124b22f99da0c55b1cc29ee963be508eb13a72" Nov 03 17:37:29.951092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:29.950995 4539 scope.go:122] "RemoveContainer" containerID="ce3bb431e76a239d18df58e13dca36b73b57341082f51def9135b36ef72c6895" Nov 03 17:37:29.965502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:29.965407 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:37:30.081048 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.081016 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:30.085982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.085955 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086005 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086036 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086059 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086086 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086144 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086192 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086252 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086307 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt\") pod \"feacadd3-6ac7-4550-9b6b-a786de858052\" (UID: \"feacadd3-6ac7-4550-9b6b-a786de858052\") " Nov 03 17:37:30.086467 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086393 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.086697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086474 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:30.086697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086605 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.086808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086747 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.086808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086749 4539 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feacadd3-6ac7-4550-9b6b-a786de858052-istiod-ca-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.086808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086788 4539 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.086808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086793 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.086808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.086805 4539 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-data\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.089368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.089344 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.089582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.089379 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:30.090128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.090067 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "kube-api-access-gjhwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:30.090248 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.090179 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo" pod "feacadd3-6ac7-4550-9b6b-a786de858052" (UID: "feacadd3-6ac7-4550-9b6b-a786de858052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:30.187709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.187675 4539 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-credential-socket\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.187709 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.187707 4539 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-workload-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.187982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.187720 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhwt\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-kube-api-access-gjhwt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.187982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.187733 4539 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feacadd3-6ac7-4550-9b6b-a786de858052-istio-envoy\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.187982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.187745 4539 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feacadd3-6ac7-4550-9b6b-a786de858052-istio-token\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.187982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.187756 4539 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feacadd3-6ac7-4550-9b6b-a786de858052-istio-podinfo\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.292582 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.292546 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:30.292811 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.292631 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:30.328006 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.327971 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:30.957954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:30.957927 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw" Nov 03 17:37:31.055092 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:31.055047 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw"] Nov 03 17:37:31.060886 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:31.060854 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ktrzw"] Nov 03 17:37:31.659592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:31.659558 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:32.106823 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:32.106789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:32.304270 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:32.304237 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:32.468350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:32.468274 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" path="/var/lib/kubelet/pods/feacadd3-6ac7-4550-9b6b-a786de858052/volumes" Nov 03 17:37:33.272260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:33.272226 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:33.671329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:33.671292 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:34.151760 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:34.151722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:35.079439 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:35.079369 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:36.675781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:36.675745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:36.819536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:36.819507 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:38.269650 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:38.269618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:39.685029 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.685003 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:37:39.719331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.719305 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:37:39.781617 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.781583 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume\") pod \"046ca71b-e9cd-4b84-acca-b294c377d0e5\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " Nov 03 17:37:39.781827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.781749 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n\" (UniqueName: \"kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n\") pod \"046ca71b-e9cd-4b84-acca-b294c377d0e5\" (UID: \"046ca71b-e9cd-4b84-acca-b294c377d0e5\") " Nov 03 17:37:39.782472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.782419 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume" pod "046ca71b-e9cd-4b84-acca-b294c377d0e5" (UID: "046ca71b-e9cd-4b84-acca-b294c377d0e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:39.782581 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.782557 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/046ca71b-e9cd-4b84-acca-b294c377d0e5-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:39.784565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.784542 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n" pod "046ca71b-e9cd-4b84-acca-b294c377d0e5" (UID: "046ca71b-e9cd-4b84-acca-b294c377d0e5"). InnerVolumeSpecName "kube-api-access-n5t2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:39.882227 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.882195 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:39.883275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.883249 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25\" (UniqueName: \"kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25\") pod \"08e7d664-d21b-4bb8-831b-fda26148cd6b\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " Nov 03 17:37:39.883409 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.883337 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume\") pod \"08e7d664-d21b-4bb8-831b-fda26148cd6b\" (UID: \"08e7d664-d21b-4bb8-831b-fda26148cd6b\") " Nov 03 17:37:39.883569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.883555 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5t2n\" (UniqueName: \"kubernetes.io/projected/046ca71b-e9cd-4b84-acca-b294c377d0e5-kube-api-access-n5t2n\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:39.883757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.883735 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume" pod "08e7d664-d21b-4bb8-831b-fda26148cd6b" (UID: "08e7d664-d21b-4bb8-831b-fda26148cd6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:39.885967 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.885942 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25" pod "08e7d664-d21b-4bb8-831b-fda26148cd6b" (UID: "08e7d664-d21b-4bb8-831b-fda26148cd6b"). InnerVolumeSpecName "kube-api-access-j6b25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:39.984794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.984709 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6b25\" (UniqueName: \"kubernetes.io/projected/08e7d664-d21b-4bb8-831b-fda26148cd6b-kube-api-access-j6b25\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:39.984794 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:39.984740 4539 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08e7d664-d21b-4bb8-831b-fda26148cd6b-plugins-config-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:37:40.007763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.007726 4539 generic.go:356] "Generic (PLEG): container finished" podID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerID="784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360" exitCode=0 Nov 03 17:37:40.007992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.007769 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" Nov 03 17:37:40.007992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.007814 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" event={"ID":"046ca71b-e9cd-4b84-acca-b294c377d0e5","Type":"ContainerDied","Data":"784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360"} Nov 03 17:37:40.007992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.007871 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c" event={"ID":"046ca71b-e9cd-4b84-acca-b294c377d0e5","Type":"ContainerDied","Data":"8a1a71a66698a0be5cb173042b9ebc7169d40b1450ce3f635088fee8c182e4e2"} Nov 03 17:37:40.007992 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.007895 4539 scope.go:122] "RemoveContainer" containerID="784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360" Nov 03 17:37:40.009332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.009309 4539 generic.go:356] "Generic (PLEG): container finished" podID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerID="a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed" exitCode=0 Nov 03 17:37:40.009495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.009348 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" event={"ID":"08e7d664-d21b-4bb8-831b-fda26148cd6b","Type":"ContainerDied","Data":"a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed"} Nov 03 17:37:40.009495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.009359 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" Nov 03 17:37:40.009495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.009382 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8" event={"ID":"08e7d664-d21b-4bb8-831b-fda26148cd6b","Type":"ContainerDied","Data":"b751780cde53f7f05334ecb203e8f7ab45d834709663148f0321dbf004151de4"} Nov 03 17:37:40.021205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.021182 4539 scope.go:122] "RemoveContainer" containerID="784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360" Nov 03 17:37:40.021652 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:37:40.021622 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360\": container with ID starting with 784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360 not found: ID does not exist" containerID="784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360" Nov 03 17:37:40.021746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.021665 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360"} err="failed to get container status \"784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360\": rpc error: code = NotFound desc = could not find container \"784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360\": container with ID starting with 784ba3b96e811bfac705e3696da45097fb29f1373797848af80725ae73f71360 not found: ID does not exist" Nov 03 17:37:40.021746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.021690 4539 scope.go:122] "RemoveContainer" containerID="a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed" Nov 03 17:37:40.030541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.030525 4539 scope.go:122] "RemoveContainer" containerID="a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed" Nov 03 17:37:40.030779 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:37:40.030760 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed\": container with ID starting with a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed not found: ID does not exist" containerID="a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed" Nov 03 17:37:40.030844 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.030785 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed"} err="failed to get container status \"a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed\": rpc error: code = NotFound desc = could not find container \"a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed\": container with ID starting with a5fc08701835034e96875433f791f560deacd4517a9b8f080fda7b554dda0aed not found: ID does not exist" Nov 03 17:37:40.061560 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.061527 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c"] Nov 03 17:37:40.073229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.073202 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e2d835-545789d8c-jwx7c"] Nov 03 17:37:40.114336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.114305 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8"] Nov 03 17:37:40.120051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.120025 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e2d835-c4445495-89rg8"] Nov 03 17:37:40.468772 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.468740 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" path="/var/lib/kubelet/pods/046ca71b-e9cd-4b84-acca-b294c377d0e5/volumes" Nov 03 17:37:40.469115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:40.469102 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" path="/var/lib/kubelet/pods/08e7d664-d21b-4bb8-831b-fda26148cd6b/volumes" Nov 03 17:37:41.451132 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:41.451075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:43.042151 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:43.042116 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:43.228017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:43.227985 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:44.523442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:44.523392 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:44.672607 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:44.672572 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:44.885527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:44.885405 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:45.052936 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:45.052902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:45.199892 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:45.199810 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:46.572015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:46.571977 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:51.077055 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:51.077022 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:56.156837 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:37:56.156802 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:37:56.881284 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:56.881246 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:58.129830 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:58.129800 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:59.186608 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:59.186572 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:37:59.344525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:37:59.344490 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:02.590557 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:02.590526 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:02.732160 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:02.732121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:08.245580 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:08.245525 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:08.394492 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:08.394409 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:08.604130 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:08.604086 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:09.805614 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:09.805576 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:10.192703 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:10.192619 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:11.385206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:11.385170 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mqxzs_dfbb04a9-f7d4-4438-9465-867b1f6ab446/manager/0.log" Nov 03 17:38:11.597558 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:11.597524 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jz5xc_9f80cd1d-c291-40d4-866b-6b1ee9005dac/manager/0.log" Nov 03 17:38:12.078484 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:12.078451 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wtqt6_a5a057da-36a9-417d-a66d-3f56cc6c6fff/manager/0.log" Nov 03 17:38:12.585511 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:12.585472 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5ldg2_6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b/manager/0.log" Nov 03 17:38:13.790077 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:13.790048 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6b4p_ff6e4b8d-1c33-4e59-9aad-512e10134470/manager/0.log" Nov 03 17:38:13.994051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:13.994020 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kswm7_22bcb2db-3d3f-49cb-b196-de4ca60476ef/manager/0.log" Nov 03 17:38:14.211926 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:14.211833 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2bgbv_a88f549e-c107-4cb0-a806-5b9dcc15c271/manager/0.log" Nov 03 17:38:14.279201 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:14.279167 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:14.432893 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:14.432865 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:14.920847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:14.920817 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5lqff_8f60d60e-9283-4377-b974-fe3e97be1bff/server/0.log" Nov 03 17:38:15.115342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:15.115313 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-625qt_e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd/manager/0.log" Nov 03 17:38:18.123109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:18.123075 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:19.068150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:19.068118 4539 scope.go:122] "RemoveContainer" containerID="a83f8694195e00bce48b9b51e3841cecad5413f5d90545951477e70be62b6795" Nov 03 17:38:20.037629 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:20.037587 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:20.524889 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:20.524855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:20.765320 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:20.765284 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:23.465591 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:23.465543 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:23.610008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:23.609974 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:24.729106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:24.729072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:24.916822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:24.916766 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:25.074656 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:25.074625 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:25.229684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:25.229651 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:26.156403 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:38:26.156365 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:38:26.658329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:26.658296 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:28.277235 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:28.277201 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:30.029941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:30.029912 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:31.805104 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:31.805068 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:33.475311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:33.475276 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:35.184120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:35.184084 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:36.782089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:36.782049 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:38.333959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:38.333929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:39.997128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:39.997098 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:41.808102 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:41.808061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:43.423522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:43.423479 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:45.081661 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:45.081620 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:46.580111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:46.580072 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:48.169706 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:48.169675 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:49.792841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:49.792803 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:50.017694 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:50.017658 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:50.207756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:50.207679 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:51.309640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:51.309605 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:51.826785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:51.826751 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:52.003989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:52.003950 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:55.816206 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:38:55.816171 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:38:56.157554 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:38:56.157461 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:39:00.809773 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:00.809678 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:00.960293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:00.960249 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:17.466069 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:17.466038 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:39:17.481781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:17.481754 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:39:18.243267 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:18.243230 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:22.428337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:22.428295 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:22.719179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:22.719030 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:26.156294 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:39:26.156254 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:39:34.024396 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:34.024360 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:35.701776 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.701744 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj"] Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702153 4539 cpu_manager.go:414] "containerMap: removing container" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702169 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702184 4539 cpu_manager.go:414] "containerMap: removing container" podUID="7181cef8-9af4-472b-bda2-8829af32400a" containerName="test-curl-11342-27034e" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702190 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7181cef8-9af4-472b-bda2-8829af32400a" containerName="test-curl-11342-27034e" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702198 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1fb3c920-1180-4861-aaa1-87df4bdc1516" containerName="main" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702203 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb3c920-1180-4861-aaa1-87df4bdc1516" containerName="main" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702217 4539 cpu_manager.go:414] "containerMap: removing container" podUID="708707d8-acb2-4982-9c2c-52e48776d590" containerName="maas-api" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702222 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708707d8-acb2-4982-9c2c-52e48776d590" containerName="maas-api" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702230 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702235 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702245 4539 cpu_manager.go:414] "containerMap: removing container" podUID="ccb45b49-1a1d-4825-beb5-4cf8f983ce97" containerName="maas-api" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702255 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb45b49-1a1d-4825-beb5-4cf8f983ce97" containerName="maas-api" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702264 4539 cpu_manager.go:414] "containerMap: removing container" podUID="400a9d3e-8c3a-4669-94a2-4642adbe2525" containerName="maas-api" Nov 03 17:39:35.702260 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702269 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400a9d3e-8c3a-4669-94a2-4642adbe2525" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702277 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702281 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702288 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702292 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702297 4539 cpu_manager.go:414] "containerMap: removing container" podUID="5c982958-c024-4297-a336-c09c6f42480b" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702301 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c982958-c024-4297-a336-c09c6f42480b" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702308 4539 cpu_manager.go:414] "containerMap: removing container" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerName="payload-processing" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702312 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerName="payload-processing" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702318 4539 cpu_manager.go:414] "containerMap: removing container" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702324 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702335 4539 cpu_manager.go:414] "containerMap: removing container" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702342 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702352 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702356 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702361 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702364 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702371 4539 cpu_manager.go:414] "containerMap: removing container" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerName="payload-pre-processing" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702374 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerName="payload-pre-processing" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702384 4539 cpu_manager.go:414] "containerMap: removing container" podUID="56322f04-5883-4048-bac9-338165b17c62" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702388 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56322f04-5883-4048-bac9-338165b17c62" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702397 4539 cpu_manager.go:414] "containerMap: removing container" podUID="c711f644-5999-424c-9af3-2f43cb718841" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702401 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c711f644-5999-424c-9af3-2f43cb718841" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702419 4539 cpu_manager.go:414] "containerMap: removing container" podUID="3d620953-d68f-4c88-b01c-d31fc541dd04" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702439 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d620953-d68f-4c88-b01c-d31fc541dd04" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702447 4539 cpu_manager.go:414] "containerMap: removing container" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702451 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702457 4539 cpu_manager.go:414] "containerMap: removing container" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702460 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702467 4539 cpu_manager.go:414] "containerMap: removing container" podUID="d3f63229-5b65-4992-ba0c-69baf3a814a4" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702472 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f63229-5b65-4992-ba0c-69baf3a814a4" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702477 4539 cpu_manager.go:414] "containerMap: removing container" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702484 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702496 4539 cpu_manager.go:414] "containerMap: removing container" podUID="fc59b182-642d-4931-8dad-2a1885caf806" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702502 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc59b182-642d-4931-8dad-2a1885caf806" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702509 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b5311ef6-beae-4a0f-a463-951e90ff1f6f" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702513 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5311ef6-beae-4a0f-a463-951e90ff1f6f" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702575 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5929f511-5e10-44ff-94d3-68c7d7a6e6cb" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702587 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="400a9d3e-8c3a-4669-94a2-4642adbe2525" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702592 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb45b49-1a1d-4825-beb5-4cf8f983ce97" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702598 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2469139-15c6-45eb-ad72-ab7f4ef38e4e" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702603 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702608 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c982958-c024-4297-a336-c09c6f42480b" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702615 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="046ca71b-e9cd-4b84-acca-b294c377d0e5" containerName="payload-processing" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702620 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702624 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bc673a-89ea-43c1-85bd-57952e0c2848" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702629 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702633 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702638 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b13c5a-b79f-40ef-98bc-b0606db9772e" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702644 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e7d664-d21b-4bb8-831b-fda26148cd6b" containerName="payload-pre-processing" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702653 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="56322f04-5883-4048-bac9-338165b17c62" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702662 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="c711f644-5999-424c-9af3-2f43cb718841" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702668 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d620953-d68f-4c88-b01c-d31fc541dd04" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702673 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e552f18-887c-49cf-88c4-f043aa0d5ddf" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702678 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7edd38-07cf-4455-ba6d-f2c502452d37" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702684 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f63229-5b65-4992-ba0c-69baf3a814a4" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702689 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6fa8fa-8315-4726-9f8e-0a54a7fb33d3" containerName="istio-proxy" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702694 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc59b182-642d-4931-8dad-2a1885caf806" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702698 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5311ef6-beae-4a0f-a463-951e90ff1f6f" containerName="maas-api" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702704 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="7181cef8-9af4-472b-bda2-8829af32400a" containerName="test-curl-11342-27034e" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702708 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb3c920-1180-4861-aaa1-87df4bdc1516" containerName="main" Nov 03 17:39:35.702756 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.702713 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="708707d8-acb2-4982-9c2c-52e48776d590" containerName="maas-api" Nov 03 17:39:35.705759 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.705739 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.735997 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.735967 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj"] Nov 03 17:39:35.750637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750602 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352815c1-13a2-49fc-8075-743b28e27b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.750792 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750650 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.750792 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750775 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.750871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750799 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352815c1-13a2-49fc-8075-743b28e27b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.750871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750817 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352815c1-13a2-49fc-8075-743b28e27b10-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.750871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750843 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.750973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750905 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.751008 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.750977 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsszc\" (UniqueName: \"kubernetes.io/projected/352815c1-13a2-49fc-8075-743b28e27b10-kube-api-access-tsszc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.751046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.751024 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852159 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852259 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852278 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352815c1-13a2-49fc-8075-743b28e27b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852292 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352815c1-13a2-49fc-8075-743b28e27b10-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852310 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852334 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852357 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsszc\" (UniqueName: \"kubernetes.io/projected/352815c1-13a2-49fc-8075-743b28e27b10-kube-api-access-tsszc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852382 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852482 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352815c1-13a2-49fc-8075-743b28e27b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852822 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852777 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.852921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.852830 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.853106 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.853084 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.853181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.853104 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.853181 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.853168 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352815c1-13a2-49fc-8075-743b28e27b10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.855212 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.855192 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352815c1-13a2-49fc-8075-743b28e27b10-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.855353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.855336 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352815c1-13a2-49fc-8075-743b28e27b10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.882032 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.881994 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352815c1-13a2-49fc-8075-743b28e27b10-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:35.882218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:35.882102 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsszc\" (UniqueName: \"kubernetes.io/projected/352815c1-13a2-49fc-8075-743b28e27b10-kube-api-access-tsszc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-drxvj\" (UID: \"352815c1-13a2-49fc-8075-743b28e27b10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:36.020395 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.020290 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:36.291770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.291739 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:36.338772 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.338743 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj"] Nov 03 17:39:36.340801 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:39:36.340764 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352815c1_13a2_49fc_8075_743b28e27b10.slice/crio-5c7b14169cbed1d37daaa63060cf0a7811272ef7984216abc5e6d56f90c20f1b WatchSource:0}: Error finding container 5c7b14169cbed1d37daaa63060cf0a7811272ef7984216abc5e6d56f90c20f1b: Status 404 returned error can't find the container with id 5c7b14169cbed1d37daaa63060cf0a7811272ef7984216abc5e6d56f90c20f1b Nov 03 17:39:36.342921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.342892 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:39:36.343018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.342977 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:39:36.343018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.343005 4539 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Nov 03 17:39:36.633448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.633348 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" event={"ID":"352815c1-13a2-49fc-8075-743b28e27b10","Type":"ContainerStarted","Data":"7b43dcfa14d1bb9547edb28c3df31b2e55827066417cb829bbbf302b161f38c5"} Nov 03 17:39:36.633448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.633410 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" event={"ID":"352815c1-13a2-49fc-8075-743b28e27b10","Type":"ContainerStarted","Data":"5c7b14169cbed1d37daaa63060cf0a7811272ef7984216abc5e6d56f90c20f1b"} Nov 03 17:39:36.680545 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:36.680483 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podStartSLOduration=1.680464788 podStartE2EDuration="1.680464788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:36.678673446 +0000 UTC m=+5120.371944466" watchObservedRunningTime="2026-11-03 17:39:36.680464788 +0000 UTC m=+5120.373735809" Nov 03 17:39:37.020757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:37.020605 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:37.026007 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:37.025953 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:37.026219 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:37.026049 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:38.024452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:38.020967 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:38.024452 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:38.021037 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:39.023969 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:39.023921 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:39.024308 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:39.024278 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:40.022929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:40.022889 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:40.023639 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:40.023604 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:40.465205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:40.465165 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:40.625799 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:40.625761 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:39:41.023917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:41.023878 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:41.024562 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:41.024520 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:42.020901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:42.020859 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:42.021231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:42.021200 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:43.021448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:43.021384 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 03 17:39:43.021901 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:43.021456 4539 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 03 17:39:44.573615 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:44.573579 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:44.574220 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:44.574033 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:44.574949 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:44.574931 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" Nov 03 17:39:56.159266 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:39:56.159222 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:39:59.967449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:59.966988 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:39:59.967449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:39:59.967064 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:40:00.620698 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:40:00.620652 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:40:00.621054 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:40:00.621005 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:40:18.480454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:40:18.480408 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:40:26.157072 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:40:26.157035 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:40:56.156798 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:40:56.156762 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:41:06.881449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.878834 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4"] Nov 03 17:41:06.881449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.879471 4539 cpu_manager.go:414] "containerMap: removing container" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" Nov 03 17:41:06.881449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.879490 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" Nov 03 17:41:06.881449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.879622 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="feacadd3-6ac7-4550-9b6b-a786de858052" containerName="istio-proxy" Nov 03 17:41:06.885110 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.885075 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:06.926114 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.926082 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-39e7fadb-kserve-self-signed-certs\"" Nov 03 17:41:06.926529 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.926507 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 17:41:06.926746 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.926730 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 17:41:06.927018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.927001 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-znvz5\"" Nov 03 17:41:06.947587 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:06.947542 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4"] Nov 03 17:41:07.022016 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.021968 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.022349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.022323 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv79l\" (UniqueName: \"kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.022544 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.022524 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.022720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.022702 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.022851 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.022835 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.022984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.022969 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.131147 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.131107 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.131579 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.131495 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.131767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.131748 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.131937 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.131919 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.132211 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.132178 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.132415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.132383 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv79l\" (UniqueName: \"kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.134046 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.133487 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.134543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.134523 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.135015 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.134997 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.147171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.147135 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.167638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.167551 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv79l\" (UniqueName: \"kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.168158 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.168133 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm\") pod \"e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.220671 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.220629 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:07.782927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:07.782860 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4"] Nov 03 17:41:07.785914 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:41:07.785885 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe5ebd17_1fba_464b_81fa_4201e25783a4.slice/crio-d9aaa6d968a1aef78f051549f9eeeef6713bb2f2f1a53ca9b1a200528914f5e2 WatchSource:0}: Error finding container d9aaa6d968a1aef78f051549f9eeeef6713bb2f2f1a53ca9b1a200528914f5e2: Status 404 returned error can't find the container with id d9aaa6d968a1aef78f051549f9eeeef6713bb2f2f1a53ca9b1a200528914f5e2 Nov 03 17:41:08.189091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:08.188995 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" event={"ID":"fe5ebd17-1fba-464b-81fa-4201e25783a4","Type":"ContainerStarted","Data":"d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e"} Nov 03 17:41:08.189091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:08.189046 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" event={"ID":"fe5ebd17-1fba-464b-81fa-4201e25783a4","Type":"ContainerStarted","Data":"d9aaa6d968a1aef78f051549f9eeeef6713bb2f2f1a53ca9b1a200528914f5e2"} Nov 03 17:41:08.222091 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:08.222049 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:08.231204 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:08.231149 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" podStartSLOduration=2.231136212 podStartE2EDuration="2.231136212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:08.230361983 +0000 UTC m=+5211.923633010" watchObservedRunningTime="2026-11-03 17:41:08.231136212 +0000 UTC m=+5211.924407233" Nov 03 17:41:08.627115 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:08.627078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:08.778194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:08.778162 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:10.938453 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:10.930476 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:17.221884 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:17.221841 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:17.240938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:17.240903 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:17.285447 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:17.285394 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:41:18.583086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:18.583038 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:26.157505 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:41:26.157463 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:41:31.735910 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:31.735878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:41.403751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:41.403715 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:41.550307 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:41.550270 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:52.926683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:52.926637 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:41:56.156627 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:41:56.156583 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:41:57.152231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.152194 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff"] Nov 03 17:41:57.156171 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.156145 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.183616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.183586 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:41:57.183616 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.183603 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:41:57.186360 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.186326 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff"] Nov 03 17:41:57.235072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.235020 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.235280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.235159 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.235280 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.235208 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvhx\" (UniqueName: \"kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.336568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.336536 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvhx\" (UniqueName: \"kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.336767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.336625 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.336767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.336675 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.337282 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.337253 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.339368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.339337 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.361600 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.361568 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvhx\" (UniqueName: \"kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx\") pod \"maas-api-revoke-sub-8b19cd46-89hff\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.470250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.470162 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:41:57.773021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:57.772997 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff"] Nov 03 17:41:57.775094 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:41:57.775063 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb63d12b_f45d_4c6b_9a27_061588712180.slice/crio-e0a82b09474da3c1a5f6a1a678bd2db3176c37ba2ec499565fecf64d50a56214 WatchSource:0}: Error finding container e0a82b09474da3c1a5f6a1a678bd2db3176c37ba2ec499565fecf64d50a56214: Status 404 returned error can't find the container with id e0a82b09474da3c1a5f6a1a678bd2db3176c37ba2ec499565fecf64d50a56214 Nov 03 17:41:58.517846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:58.517714 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" event={"ID":"cb63d12b-f45d-4c6b-9a27-061588712180","Type":"ContainerStarted","Data":"d724dca2f5b414df3eb06791c7c4fa0b00e8b7d30eb76cd4c5a1aff02acf8b48"} Nov 03 17:41:58.517846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:58.517771 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" event={"ID":"cb63d12b-f45d-4c6b-9a27-061588712180","Type":"ContainerStarted","Data":"e0a82b09474da3c1a5f6a1a678bd2db3176c37ba2ec499565fecf64d50a56214"} Nov 03 17:41:58.563720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:58.563629 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" podStartSLOduration=1.563610396 podStartE2EDuration="1.563610396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:58.561243133 +0000 UTC m=+5262.254514155" watchObservedRunningTime="2026-11-03 17:41:58.563610396 +0000 UTC m=+5262.256881419" Nov 03 17:41:59.524724 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:59.524683 4539 generic.go:356] "Generic (PLEG): container finished" podID="cb63d12b-f45d-4c6b-9a27-061588712180" containerID="d724dca2f5b414df3eb06791c7c4fa0b00e8b7d30eb76cd4c5a1aff02acf8b48" exitCode=0 Nov 03 17:41:59.525202 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:41:59.524761 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" event={"ID":"cb63d12b-f45d-4c6b-9a27-061588712180","Type":"ContainerDied","Data":"d724dca2f5b414df3eb06791c7c4fa0b00e8b7d30eb76cd4c5a1aff02acf8b48"} Nov 03 17:42:00.673798 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.673773 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:42:00.768889 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.768853 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca\") pod \"cb63d12b-f45d-4c6b-9a27-061588712180\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " Nov 03 17:42:00.769076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.768991 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca\") pod \"cb63d12b-f45d-4c6b-9a27-061588712180\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " Nov 03 17:42:00.769076 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.769030 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx\" (UniqueName: \"kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx\") pod \"cb63d12b-f45d-4c6b-9a27-061588712180\" (UID: \"cb63d12b-f45d-4c6b-9a27-061588712180\") " Nov 03 17:42:00.769275 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.769244 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca" pod "cb63d12b-f45d-4c6b-9a27-061588712180" (UID: "cb63d12b-f45d-4c6b-9a27-061588712180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:00.771279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.771257 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca" pod "cb63d12b-f45d-4c6b-9a27-061588712180" (UID: "cb63d12b-f45d-4c6b-9a27-061588712180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:00.771436 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.771400 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx" pod "cb63d12b-f45d-4c6b-9a27-061588712180" (UID: "cb63d12b-f45d-4c6b-9a27-061588712180"). InnerVolumeSpecName "kube-api-access-7zvhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:00.870472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.870334 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:00.870472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.870374 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvhx\" (UniqueName: \"kubernetes.io/projected/cb63d12b-f45d-4c6b-9a27-061588712180-kube-api-access-7zvhx\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:00.870472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:00.870387 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb63d12b-f45d-4c6b-9a27-061588712180-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:01.536322 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:01.536288 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" Nov 03 17:42:01.536528 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:01.536320 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff" event={"ID":"cb63d12b-f45d-4c6b-9a27-061588712180","Type":"ContainerDied","Data":"e0a82b09474da3c1a5f6a1a678bd2db3176c37ba2ec499565fecf64d50a56214"} Nov 03 17:42:01.536528 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:01.536367 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0a82b09474da3c1a5f6a1a678bd2db3176c37ba2ec499565fecf64d50a56214" Nov 03 17:42:08.945292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:08.945240 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4"] Nov 03 17:42:08.975565 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:08.975412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:09.119602 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:09.119569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:11.810354 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.810316 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp"] Nov 03 17:42:11.810946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.810919 4539 cpu_manager.go:414] "containerMap: removing container" podUID="cb63d12b-f45d-4c6b-9a27-061588712180" containerName="revoke-keys" Nov 03 17:42:11.810946 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.810943 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb63d12b-f45d-4c6b-9a27-061588712180" containerName="revoke-keys" Nov 03 17:42:11.811332 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.811092 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb63d12b-f45d-4c6b-9a27-061588712180" containerName="revoke-keys" Nov 03 17:42:11.814226 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.814202 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.840589 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.840555 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:42:11.840770 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.840626 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:42:11.844251 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.844211 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp"] Nov 03 17:42:11.880927 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.880894 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm6z\" (UniqueName: \"kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.881128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.880943 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.881128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.881062 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.982241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.982198 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.982472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.982271 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm6z\" (UniqueName: \"kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.982472 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.982307 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.983018 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.982995 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:11.984828 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:11.984802 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:12.006210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:12.006174 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm6z\" (UniqueName: \"kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z\") pod \"maas-api-revoke-sub-7f668a8b-g6ktp\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:12.132567 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:12.132481 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:12.433751 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:12.433722 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp"] Nov 03 17:42:12.435186 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:42:12.435152 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d20866_8b2f_44e8_8579_0cae266cb3d6.slice/crio-9772fc972da476a24b3f8dc4659db9e3c6bba2123bb6e2e15a6bc320741ba03f WatchSource:0}: Error finding container 9772fc972da476a24b3f8dc4659db9e3c6bba2123bb6e2e15a6bc320741ba03f: Status 404 returned error can't find the container with id 9772fc972da476a24b3f8dc4659db9e3c6bba2123bb6e2e15a6bc320741ba03f Nov 03 17:42:12.608449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:12.600622 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" event={"ID":"b8d20866-8b2f-44e8-8579-0cae266cb3d6","Type":"ContainerStarted","Data":"9772fc972da476a24b3f8dc4659db9e3c6bba2123bb6e2e15a6bc320741ba03f"} Nov 03 17:42:13.626450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:13.621465 4539 generic.go:356] "Generic (PLEG): container finished" podID="b8d20866-8b2f-44e8-8579-0cae266cb3d6" containerID="8dc6a28f6fdee9618f4bda12800cacefc45ce758eeb710218f504cf2c2dc39b5" exitCode=0 Nov 03 17:42:13.626450 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:13.621725 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" event={"ID":"b8d20866-8b2f-44e8-8579-0cae266cb3d6","Type":"ContainerDied","Data":"8dc6a28f6fdee9618f4bda12800cacefc45ce758eeb710218f504cf2c2dc39b5"} Nov 03 17:42:14.981571 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:14.981530 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:42:14.994702 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:14.994487 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7dnkg" podUID="929514d9-b731-4a30-8a67-6d1522c756cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:42:15.020769 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.020720 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:15.095099 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.095031 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z\" (UniqueName: \"kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z\") pod \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " Nov 03 17:42:15.095303 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.095166 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca\") pod \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " Nov 03 17:42:15.095353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.095326 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca\") pod \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\" (UID: \"b8d20866-8b2f-44e8-8579-0cae266cb3d6\") " Nov 03 17:42:15.095805 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.095773 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca" pod "b8d20866-8b2f-44e8-8579-0cae266cb3d6" (UID: "b8d20866-8b2f-44e8-8579-0cae266cb3d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:15.098134 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.098093 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca" pod "b8d20866-8b2f-44e8-8579-0cae266cb3d6" (UID: "b8d20866-8b2f-44e8-8579-0cae266cb3d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:15.098417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.098391 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z" pod "b8d20866-8b2f-44e8-8579-0cae266cb3d6" (UID: "b8d20866-8b2f-44e8-8579-0cae266cb3d6"). InnerVolumeSpecName "kube-api-access-xxm6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:15.197215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.197121 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxm6z\" (UniqueName: \"kubernetes.io/projected/b8d20866-8b2f-44e8-8579-0cae266cb3d6-kube-api-access-xxm6z\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:15.197792 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.197671 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:15.198003 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.197983 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d20866-8b2f-44e8-8579-0cae266cb3d6-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:15.644555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.644519 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" Nov 03 17:42:15.644555 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.644547 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp" event={"ID":"b8d20866-8b2f-44e8-8579-0cae266cb3d6","Type":"ContainerDied","Data":"9772fc972da476a24b3f8dc4659db9e3c6bba2123bb6e2e15a6bc320741ba03f"} Nov 03 17:42:15.644784 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:15.644596 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9772fc972da476a24b3f8dc4659db9e3c6bba2123bb6e2e15a6bc320741ba03f" Nov 03 17:42:18.687775 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:18.687742 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:23.790774 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:23.790735 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:23.972416 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:23.972379 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" podUID="fe5ebd17-1fba-464b-81fa-4201e25783a4" containerName="main" containerID="cri-o://d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e" gracePeriod=45 Nov 03 17:42:24.242572 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.242539 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:42:24.313671 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.313638 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs\") pod \"fe5ebd17-1fba-464b-81fa-4201e25783a4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " Nov 03 17:42:24.313671 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.313679 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm\") pod \"fe5ebd17-1fba-464b-81fa-4201e25783a4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " Nov 03 17:42:24.313921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.313720 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache\") pod \"fe5ebd17-1fba-464b-81fa-4201e25783a4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " Nov 03 17:42:24.313921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.313787 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l\" (UniqueName: \"kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l\") pod \"fe5ebd17-1fba-464b-81fa-4201e25783a4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " Nov 03 17:42:24.313921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.313874 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home\") pod \"fe5ebd17-1fba-464b-81fa-4201e25783a4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " Nov 03 17:42:24.314109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.313933 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir\") pod \"fe5ebd17-1fba-464b-81fa-4201e25783a4\" (UID: \"fe5ebd17-1fba-464b-81fa-4201e25783a4\") " Nov 03 17:42:24.314109 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.314076 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache" pod "fe5ebd17-1fba-464b-81fa-4201e25783a4" (UID: "fe5ebd17-1fba-464b-81fa-4201e25783a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:42:24.314264 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.314208 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home" pod "fe5ebd17-1fba-464b-81fa-4201e25783a4" (UID: "fe5ebd17-1fba-464b-81fa-4201e25783a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:42:24.314358 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.314263 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir" pod "fe5ebd17-1fba-464b-81fa-4201e25783a4" (UID: "fe5ebd17-1fba-464b-81fa-4201e25783a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:42:24.314358 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.314348 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:24.314495 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.314369 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:24.316294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.316269 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l" pod "fe5ebd17-1fba-464b-81fa-4201e25783a4" (UID: "fe5ebd17-1fba-464b-81fa-4201e25783a4"). InnerVolumeSpecName "kube-api-access-tv79l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:24.316585 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.316564 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm" pod "fe5ebd17-1fba-464b-81fa-4201e25783a4" (UID: "fe5ebd17-1fba-464b-81fa-4201e25783a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:42:24.316659 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.316569 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs" pod "fe5ebd17-1fba-464b-81fa-4201e25783a4" (UID: "fe5ebd17-1fba-464b-81fa-4201e25783a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:24.360879 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.360774 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:24.418577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.418462 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv79l\" (UniqueName: \"kubernetes.io/projected/fe5ebd17-1fba-464b-81fa-4201e25783a4-kube-api-access-tv79l\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:24.418577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.418503 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:24.418577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.418518 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5ebd17-1fba-464b-81fa-4201e25783a4-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:24.418577 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.418530 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5ebd17-1fba-464b-81fa-4201e25783a4-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:42:24.512569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.512532 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:24.710553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.710463 4539 generic.go:356] "Generic (PLEG): container finished" podID="fe5ebd17-1fba-464b-81fa-4201e25783a4" containerID="d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e" exitCode=0 Nov 03 17:42:24.710553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.710501 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" event={"ID":"fe5ebd17-1fba-464b-81fa-4201e25783a4","Type":"ContainerDied","Data":"d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e"} Nov 03 17:42:24.710553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.710532 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" event={"ID":"fe5ebd17-1fba-464b-81fa-4201e25783a4","Type":"ContainerDied","Data":"d9aaa6d968a1aef78f051549f9eeeef6713bb2f2f1a53ca9b1a200528914f5e2"} Nov 03 17:42:24.710553 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.710549 4539 scope.go:122] "RemoveContainer" containerID="d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e" Nov 03 17:42:24.711051 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.710549 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4" Nov 03 17:42:24.722797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.722777 4539 scope.go:122] "RemoveContainer" containerID="d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e" Nov 03 17:42:24.723108 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:42:24.723085 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e\": container with ID starting with d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e not found: ID does not exist" containerID="d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e" Nov 03 17:42:24.723176 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.723118 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e"} err="failed to get container status \"d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e\": rpc error: code = NotFound desc = could not find container \"d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e\": container with ID starting with d4e5673a010caf249b69cb22fd02fa00f31bade33bc45a7355c3477e24170b6e not found: ID does not exist" Nov 03 17:42:24.762229 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.762197 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4"] Nov 03 17:42:24.773741 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:24.773714 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-6f5fddc7f6-2tpv4"] Nov 03 17:42:26.156962 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:42:26.156919 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:42:26.476373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:26.476280 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5ebd17-1fba-464b-81fa-4201e25783a4" path="/var/lib/kubelet/pods/fe5ebd17-1fba-464b-81fa-4201e25783a4/volumes" Nov 03 17:42:27.825674 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:27.825636 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:28.006534 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:28.006498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:29.103973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:29.103937 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:29.370233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:29.370155 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:29.522433 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:29.522387 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:31.215250 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:31.215205 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:33.006480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:33.006445 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:34.721644 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:34.721603 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:36.517536 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:36.517498 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:38.312349 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:38.312317 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:40.201880 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:40.201399 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:41.994039 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:41.994000 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:43.320945 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:43.320905 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:43.494755 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:43.494719 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:43.648594 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:43.648508 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:43.815983 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:43.815942 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:45.427383 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.427346 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg"] Nov 03 17:42:45.428025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.428005 4539 cpu_manager.go:414] "containerMap: removing container" podUID="b8d20866-8b2f-44e8-8579-0cae266cb3d6" containerName="revoke-keys" Nov 03 17:42:45.428078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.428029 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d20866-8b2f-44e8-8579-0cae266cb3d6" containerName="revoke-keys" Nov 03 17:42:45.428078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.428055 4539 cpu_manager.go:414] "containerMap: removing container" podUID="fe5ebd17-1fba-464b-81fa-4201e25783a4" containerName="main" Nov 03 17:42:45.428078 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.428064 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5ebd17-1fba-464b-81fa-4201e25783a4" containerName="main" Nov 03 17:42:45.428180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.428170 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d20866-8b2f-44e8-8579-0cae266cb3d6" containerName="revoke-keys" Nov 03 17:42:45.428222 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.428185 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5ebd17-1fba-464b-81fa-4201e25783a4" containerName="main" Nov 03 17:42:45.431843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.431819 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.452298 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.452266 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4b1292\"/\"default-dockercfg-jvhzs\"" Nov 03 17:42:45.464718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.464690 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4b1292\"/\"openshift-service-ca.crt\"" Nov 03 17:42:45.464718 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.464699 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4b1292\"/\"simulator-single-main-4b1292-kserve-self-signed-certs\"" Nov 03 17:42:45.464973 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.464791 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4b1292\"/\"kube-root-ca.crt\"" Nov 03 17:42:45.467338 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.467312 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg"] Nov 03 17:42:45.514060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.514015 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.514060 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.514063 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.514344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.514135 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.514344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.514206 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.514344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.514253 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.514344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.514335 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzlj\" (UniqueName: \"kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.531344 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.531297 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:45.615432 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615396 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.615637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615493 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzlj\" (UniqueName: \"kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.615637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615543 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.615637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615562 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.615637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615584 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.615637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615632 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.615943 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615914 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.616017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.615956 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.616128 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.616100 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.617938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.617920 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.618246 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.618228 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.639340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.639309 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzlj\" (UniqueName: \"kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj\") pod \"simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.756961 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.756852 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:45.771514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:45.771484 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:46.060233 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:46.060207 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg"] Nov 03 17:42:46.062571 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:42:46.062545 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1096520_9b51_40b1_88bc_6a679a7268ce.slice/crio-a1202e0114a500cf4900eb2db38fde0ea536fc76d6d069ea3e552f03f643472d WatchSource:0}: Error finding container a1202e0114a500cf4900eb2db38fde0ea536fc76d6d069ea3e552f03f643472d: Status 404 returned error can't find the container with id a1202e0114a500cf4900eb2db38fde0ea536fc76d6d069ea3e552f03f643472d Nov 03 17:42:46.838787 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:46.838745 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" event={"ID":"f1096520-9b51-40b1-88bc-6a679a7268ce","Type":"ContainerStarted","Data":"71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8"} Nov 03 17:42:46.839257 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:46.838793 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" event={"ID":"f1096520-9b51-40b1-88bc-6a679a7268ce","Type":"ContainerStarted","Data":"a1202e0114a500cf4900eb2db38fde0ea536fc76d6d069ea3e552f03f643472d"} Nov 03 17:42:46.883326 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:46.883265 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" podStartSLOduration=1.883250323 podStartE2EDuration="1.883250323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:46.880847426 +0000 UTC m=+5310.574118454" watchObservedRunningTime="2026-11-03 17:42:46.883250323 +0000 UTC m=+5310.576521346" Nov 03 17:42:47.614871 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:47.614832 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:47.757448 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:47.757383 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:49.531976 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:49.531943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:51.205093 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:51.205051 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:52.989819 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:52.989776 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:54.686808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:54.686766 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:55.758170 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:55.758131 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:55.770958 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:55.770932 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:55.901305 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:55.901274 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:42:56.156979 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:42:56.156941 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:42:56.394113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:56.394078 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:58.195813 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:58.195779 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:42:59.895312 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:42:59.895281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:01.590603 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:01.590563 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:03.399921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:03.399878 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:05.201400 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:05.201367 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:06.948615 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:06.948574 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:08.633807 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:08.633768 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:10.403162 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:10.403121 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:12.184084 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:12.184043 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:13.894193 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:13.894143 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:15.583640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:15.583603 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:17.385156 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:17.385118 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:18.800780 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:18.800748 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:19.092089 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:19.091993 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:20.961180 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:20.961131 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:22.817937 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:22.817903 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:24.510244 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:24.510211 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:26.156665 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:43:26.156626 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:43:26.160782 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:26.160754 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:26.311568 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:26.311524 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:26.697489 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:26.697455 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:28.098649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:28.098610 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:29.905985 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:29.905951 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:31.082119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:31.082084 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg"] Nov 03 17:43:31.256704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:31.256664 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:31.797689 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:31.797659 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:32.410524 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:32.410493 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:32.566342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:32.566309 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:33.440767 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:33.440732 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:33.616210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:33.616177 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:39.636917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:39.636876 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:39.799781 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:39.799745 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:43:46.106370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.106299 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" podUID="f1096520-9b51-40b1-88bc-6a679a7268ce" containerName="main" containerID="cri-o://71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8" gracePeriod=45 Nov 03 17:43:46.363465 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.363404 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:43:46.526022 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.525988 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm\") pod \"f1096520-9b51-40b1-88bc-6a679a7268ce\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " Nov 03 17:43:46.526218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526036 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home\") pod \"f1096520-9b51-40b1-88bc-6a679a7268ce\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " Nov 03 17:43:46.526218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526099 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs\") pod \"f1096520-9b51-40b1-88bc-6a679a7268ce\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " Nov 03 17:43:46.526218 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526169 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj\" (UniqueName: \"kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj\") pod \"f1096520-9b51-40b1-88bc-6a679a7268ce\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " Nov 03 17:43:46.526401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526247 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir\") pod \"f1096520-9b51-40b1-88bc-6a679a7268ce\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " Nov 03 17:43:46.526401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526276 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache\") pod \"f1096520-9b51-40b1-88bc-6a679a7268ce\" (UID: \"f1096520-9b51-40b1-88bc-6a679a7268ce\") " Nov 03 17:43:46.526401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526331 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home" pod "f1096520-9b51-40b1-88bc-6a679a7268ce" (UID: "f1096520-9b51-40b1-88bc-6a679a7268ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:43:46.526597 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526491 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir" pod "f1096520-9b51-40b1-88bc-6a679a7268ce" (UID: "f1096520-9b51-40b1-88bc-6a679a7268ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:43:46.526662 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526586 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache" pod "f1096520-9b51-40b1-88bc-6a679a7268ce" (UID: "f1096520-9b51-40b1-88bc-6a679a7268ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:43:46.526765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526658 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:43:46.526868 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.526855 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:43:46.528558 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.528532 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm" pod "f1096520-9b51-40b1-88bc-6a679a7268ce" (UID: "f1096520-9b51-40b1-88bc-6a679a7268ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:43:46.528683 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.528625 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs" pod "f1096520-9b51-40b1-88bc-6a679a7268ce" (UID: "f1096520-9b51-40b1-88bc-6a679a7268ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:43:46.528729 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.528706 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj" pod "f1096520-9b51-40b1-88bc-6a679a7268ce" (UID: "f1096520-9b51-40b1-88bc-6a679a7268ce"). InnerVolumeSpecName "kube-api-access-2jzlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:46.628369 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.628226 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:43:46.628369 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.628257 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1096520-9b51-40b1-88bc-6a679a7268ce-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:43:46.628369 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.628267 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jzlj\" (UniqueName: \"kubernetes.io/projected/f1096520-9b51-40b1-88bc-6a679a7268ce-kube-api-access-2jzlj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:43:46.628369 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:46.628277 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1096520-9b51-40b1-88bc-6a679a7268ce-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:43:47.186868 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.186830 4539 generic.go:356] "Generic (PLEG): container finished" podID="f1096520-9b51-40b1-88bc-6a679a7268ce" containerID="71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8" exitCode=0 Nov 03 17:43:47.187324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.186880 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" Nov 03 17:43:47.187324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.186911 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" event={"ID":"f1096520-9b51-40b1-88bc-6a679a7268ce","Type":"ContainerDied","Data":"71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8"} Nov 03 17:43:47.187324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.186957 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg" event={"ID":"f1096520-9b51-40b1-88bc-6a679a7268ce","Type":"ContainerDied","Data":"a1202e0114a500cf4900eb2db38fde0ea536fc76d6d069ea3e552f03f643472d"} Nov 03 17:43:47.187324 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.186976 4539 scope.go:122] "RemoveContainer" containerID="71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8" Nov 03 17:43:47.196909 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.196890 4539 scope.go:122] "RemoveContainer" containerID="71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8" Nov 03 17:43:47.197197 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:43:47.197179 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8\": container with ID starting with 71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8 not found: ID does not exist" containerID="71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8" Nov 03 17:43:47.197248 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.197205 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8"} err="failed to get container status \"71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8\": rpc error: code = NotFound desc = could not find container \"71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8\": container with ID starting with 71f01879dfec93989739e3a81ac1baf0ae068837fe0cca36fe8d1f6694a175b8 not found: ID does not exist" Nov 03 17:43:47.243342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.243309 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg"] Nov 03 17:43:47.255094 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:47.255059 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4b1292/simulator-single-main-4b1292-kserve-6df98bc5c6-hrhcg"] Nov 03 17:43:48.467335 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:43:48.467303 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1096520-9b51-40b1-88bc-6a679a7268ce" path="/var/lib/kubelet/pods/f1096520-9b51-40b1-88bc-6a679a7268ce/volumes" Nov 03 17:43:56.157896 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:43:56.157851 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:44:04.136690 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:04.136650 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:04.312979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:04.312943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:05.407311 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:05.407273 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:05.640079 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:05.640036 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:05.801207 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:05.801172 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:05.948605 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:05.948566 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:07.300114 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:07.300077 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:07.523107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:07.523073 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:08.724832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:08.724795 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:09.360646 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:09.360611 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:11.258228 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:11.258189 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:13.001477 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:13.001298 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:14.712642 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:14.712608 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:14.865777 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:14.865744 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:16.495402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:16.495368 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:17.503012 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:17.502981 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:44:17.523832 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:17.523798 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:44:18.294652 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:18.294618 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:18.926240 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:18.926208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:20.006353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:20.006308 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:21.822831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:21.822788 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:23.612385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:23.612348 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:25.089474 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:25.089415 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:25.237941 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:25.237892 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:25.387872 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:25.387789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:26.156942 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:44:26.156899 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:44:26.747412 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:26.747382 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:27.283933 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:27.283899 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:27.572293 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:27.572208 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:27.873930 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:27.873836 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:32.223880 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.223842 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68"] Nov 03 17:44:32.224539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.224518 4539 cpu_manager.go:414] "containerMap: removing container" podUID="f1096520-9b51-40b1-88bc-6a679a7268ce" containerName="main" Nov 03 17:44:32.224658 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.224541 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1096520-9b51-40b1-88bc-6a679a7268ce" containerName="main" Nov 03 17:44:32.224719 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.224660 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1096520-9b51-40b1-88bc-6a679a7268ce" containerName="main" Nov 03 17:44:32.229677 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.229651 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.245025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.244997 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 17:44:32.256417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.256393 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 17:44:32.257045 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.257019 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-znvz5\"" Nov 03 17:44:32.257203 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.257094 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4749b850-kserve-self-signed-certs\"" Nov 03 17:44:32.259145 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.259122 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68"] Nov 03 17:44:32.325438 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.325375 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:32.350558 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.350528 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.350765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.350597 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.350765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.350678 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.350765 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.350711 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.350960 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.350807 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9wb\" (UniqueName: \"kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.350960 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.350855 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452061 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452030 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452085 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9wb\" (UniqueName: \"kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452116 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452163 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452234 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452618 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452270 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452615 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452616 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.452676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.452660 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.454712 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.454688 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.454955 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.454937 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.477126 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.477059 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9wb\" (UniqueName: \"kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb\") pod \"e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.557619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.557582 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:32.871550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:32.871286 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68"] Nov 03 17:44:32.873717 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:44:32.873687 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68bda8e6_f896_4ada_85b8_0178831756c5.slice/crio-fc7fd88e925ae26e26c731c52c6c0ba596428425ab8bc57d39d00bb406b16736 WatchSource:0}: Error finding container fc7fd88e925ae26e26c731c52c6c0ba596428425ab8bc57d39d00bb406b16736: Status 404 returned error can't find the container with id fc7fd88e925ae26e26c731c52c6c0ba596428425ab8bc57d39d00bb406b16736 Nov 03 17:44:33.440350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:33.440309 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" event={"ID":"68bda8e6-f896-4ada-85b8-0178831756c5","Type":"ContainerStarted","Data":"4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0"} Nov 03 17:44:33.440350 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:33.440365 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" event={"ID":"68bda8e6-f896-4ada-85b8-0178831756c5","Type":"ContainerStarted","Data":"fc7fd88e925ae26e26c731c52c6c0ba596428425ab8bc57d39d00bb406b16736"} Nov 03 17:44:33.557836 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:33.557798 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:42.558241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:42.558199 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:42.571277 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:42.571248 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:42.615897 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:42.615829 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" podStartSLOduration=10.615810797 podStartE2EDuration="10.615810797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:33.491644035 +0000 UTC m=+5417.184915068" watchObservedRunningTime="2026-11-03 17:44:42.615810797 +0000 UTC m=+5426.309081823" Nov 03 17:44:43.544720 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:43.544685 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:44:49.583753 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:49.583690 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:51.182302 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:51.182267 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:51.334632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:44:51.334593 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:44:56.155930 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:44:56.155887 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:45:06.300758 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.300663 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw"] Nov 03 17:45:06.308540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.308508 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.323954 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.323930 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:45:06.335797 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.335769 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:45:06.338259 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.338229 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw"] Nov 03 17:45:06.409915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.409876 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.410142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.409929 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqdf\" (UniqueName: \"kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.410142 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.409954 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.511331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.511293 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqdf\" (UniqueName: \"kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.511331 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.511333 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.511588 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.511454 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.512088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.512065 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.514041 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.514019 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.534704 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.534678 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqdf\" (UniqueName: \"kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf\") pod \"maas-api-revoke-sub-452b6d7b-sglcw\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.622406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.622316 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:06.927785 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:06.927760 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw"] Nov 03 17:45:06.930235 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:45:06.930205 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f39a3df_801f_4386_924e_81e61157e2df.slice/crio-f9a41336c82d9c6d1586d598721f1621ea39d9c109862e7d165b8695df84d362 WatchSource:0}: Error finding container f9a41336c82d9c6d1586d598721f1621ea39d9c109862e7d165b8695df84d362: Status 404 returned error can't find the container with id f9a41336c82d9c6d1586d598721f1621ea39d9c109862e7d165b8695df84d362 Nov 03 17:45:07.708196 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:07.708151 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" event={"ID":"1f39a3df-801f-4386-924e-81e61157e2df","Type":"ContainerStarted","Data":"6359f9a50d8106efcdf188fb8917e578a5786a665bad8969bf1124ba845b6fdf"} Nov 03 17:45:07.708757 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:07.708222 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" event={"ID":"1f39a3df-801f-4386-924e-81e61157e2df","Type":"ContainerStarted","Data":"f9a41336c82d9c6d1586d598721f1621ea39d9c109862e7d165b8695df84d362"} Nov 03 17:45:07.752264 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:07.752207 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" podStartSLOduration=1.75219198 podStartE2EDuration="1.75219198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:07.751409047 +0000 UTC m=+5451.444680069" watchObservedRunningTime="2026-11-03 17:45:07.75219198 +0000 UTC m=+5451.445463001" Nov 03 17:45:08.716401 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:08.716362 4539 generic.go:356] "Generic (PLEG): container finished" podID="1f39a3df-801f-4386-924e-81e61157e2df" containerID="6359f9a50d8106efcdf188fb8917e578a5786a665bad8969bf1124ba845b6fdf" exitCode=0 Nov 03 17:45:08.717000 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:08.716454 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" event={"ID":"1f39a3df-801f-4386-924e-81e61157e2df","Type":"ContainerDied","Data":"6359f9a50d8106efcdf188fb8917e578a5786a665bad8969bf1124ba845b6fdf"} Nov 03 17:45:10.183088 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.182735 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:10.295449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.292040 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf\" (UniqueName: \"kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf\") pod \"1f39a3df-801f-4386-924e-81e61157e2df\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " Nov 03 17:45:10.295449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.292114 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca\") pod \"1f39a3df-801f-4386-924e-81e61157e2df\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " Nov 03 17:45:10.295449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.292206 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca\") pod \"1f39a3df-801f-4386-924e-81e61157e2df\" (UID: \"1f39a3df-801f-4386-924e-81e61157e2df\") " Nov 03 17:45:10.295449 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.294844 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca" pod "1f39a3df-801f-4386-924e-81e61157e2df" (UID: "1f39a3df-801f-4386-924e-81e61157e2df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:10.298019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.297699 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca" pod "1f39a3df-801f-4386-924e-81e61157e2df" (UID: "1f39a3df-801f-4386-924e-81e61157e2df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:10.298019 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.297960 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf" pod "1f39a3df-801f-4386-924e-81e61157e2df" (UID: "1f39a3df-801f-4386-924e-81e61157e2df"). InnerVolumeSpecName "kube-api-access-wrqdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:10.393542 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.393447 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqdf\" (UniqueName: \"kubernetes.io/projected/1f39a3df-801f-4386-924e-81e61157e2df-kube-api-access-wrqdf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:10.393542 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.393479 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-openshift-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:10.393542 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.393490 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f39a3df-801f-4386-924e-81e61157e2df-maas-api-xks-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:10.759537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.759015 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" event={"ID":"1f39a3df-801f-4386-924e-81e61157e2df","Type":"ContainerDied","Data":"f9a41336c82d9c6d1586d598721f1621ea39d9c109862e7d165b8695df84d362"} Nov 03 17:45:10.759537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.759128 4539 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a41336c82d9c6d1586d598721f1621ea39d9c109862e7d165b8695df84d362" Nov 03 17:45:10.759537 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:10.759203 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw" Nov 03 17:45:17.564711 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:17.564673 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68"] Nov 03 17:45:17.679210 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:17.679175 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:19.063442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:19.063380 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:23.132500 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:23.132468 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:26.159435 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:45:26.159377 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:45:28.977342 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:28.977274 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:32.585541 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.585470 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" podUID="68bda8e6-f896-4ada-85b8-0178831756c5" containerName="main" containerID="cri-o://4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0" gracePeriod=45 Nov 03 17:45:32.821970 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.821939 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:32.845312 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.845245 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:45:32.913319 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913288 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir\") pod \"68bda8e6-f896-4ada-85b8-0178831756c5\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " Nov 03 17:45:32.913547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913369 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache\") pod \"68bda8e6-f896-4ada-85b8-0178831756c5\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " Nov 03 17:45:32.913547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913407 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb\" (UniqueName: \"kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb\") pod \"68bda8e6-f896-4ada-85b8-0178831756c5\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " Nov 03 17:45:32.913547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913449 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs\") pod \"68bda8e6-f896-4ada-85b8-0178831756c5\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " Nov 03 17:45:32.913547 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913503 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home\") pod \"68bda8e6-f896-4ada-85b8-0178831756c5\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " Nov 03 17:45:32.913858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913565 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm\") pod \"68bda8e6-f896-4ada-85b8-0178831756c5\" (UID: \"68bda8e6-f896-4ada-85b8-0178831756c5\") " Nov 03 17:45:32.913858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913658 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir" pod "68bda8e6-f896-4ada-85b8-0178831756c5" (UID: "68bda8e6-f896-4ada-85b8-0178831756c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:32.913858 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913722 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache" pod "68bda8e6-f896-4ada-85b8-0178831756c5" (UID: "68bda8e6-f896-4ada-85b8-0178831756c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:32.913980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913917 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:32.913980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.913937 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:32.914098 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.914056 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home" pod "68bda8e6-f896-4ada-85b8-0178831756c5" (UID: "68bda8e6-f896-4ada-85b8-0178831756c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:32.916135 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.916104 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb" pod "68bda8e6-f896-4ada-85b8-0178831756c5" (UID: "68bda8e6-f896-4ada-85b8-0178831756c5"). InnerVolumeSpecName "kube-api-access-dq9wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:32.916282 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.916126 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs" pod "68bda8e6-f896-4ada-85b8-0178831756c5" (UID: "68bda8e6-f896-4ada-85b8-0178831756c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:32.916282 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.916222 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm" pod "68bda8e6-f896-4ada-85b8-0178831756c5" (UID: "68bda8e6-f896-4ada-85b8-0178831756c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:32.922161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.922139 4539 generic.go:356] "Generic (PLEG): container finished" podID="68bda8e6-f896-4ada-85b8-0178831756c5" containerID="4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0" exitCode=0 Nov 03 17:45:32.922241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.922177 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" event={"ID":"68bda8e6-f896-4ada-85b8-0178831756c5","Type":"ContainerDied","Data":"4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0"} Nov 03 17:45:32.922241 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.922234 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" event={"ID":"68bda8e6-f896-4ada-85b8-0178831756c5","Type":"ContainerDied","Data":"fc7fd88e925ae26e26c731c52c6c0ba596428425ab8bc57d39d00bb406b16736"} Nov 03 17:45:32.922327 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.922257 4539 scope.go:122] "RemoveContainer" containerID="4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0" Nov 03 17:45:32.922327 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.922195 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68" Nov 03 17:45:32.935632 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.935606 4539 scope.go:122] "RemoveContainer" containerID="4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0" Nov 03 17:45:32.935959 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:45:32.935939 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0\": container with ID starting with 4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0 not found: ID does not exist" containerID="4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0" Nov 03 17:45:32.936025 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.935971 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0"} err="failed to get container status \"4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0\": rpc error: code = NotFound desc = could not find container \"4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0\": container with ID starting with 4a6fa02715f3f6dad8cdff6e18f2f30b2dd6a8f0c99cf1b943d96b723718d2e0 not found: ID does not exist" Nov 03 17:45:32.968017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.967990 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:32.972928 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.972901 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68"] Nov 03 17:45:32.977417 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:32.977394 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4749b850-kserve-7d6bd6d957-h8x68"] Nov 03 17:45:33.015021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:33.014989 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9wb\" (UniqueName: \"kubernetes.io/projected/68bda8e6-f896-4ada-85b8-0178831756c5-kube-api-access-dq9wb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:33.015021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:33.015016 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bda8e6-f896-4ada-85b8-0178831756c5-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:33.015021 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:33.015028 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:33.015249 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:33.015035 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bda8e6-f896-4ada-85b8-0178831756c5-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:45:34.466930 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:45:34.466893 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bda8e6-f896-4ada-85b8-0178831756c5" path="/var/lib/kubelet/pods/68bda8e6-f896-4ada-85b8-0178831756c5/volumes" Nov 03 17:45:56.156142 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:45:56.156102 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:46:19.181345 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:46:19.181318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:26.156745 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:46:26.156704 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:46:44.245808 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:46:44.245709 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:44.393530 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:46:44.393493 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:45.158111 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:46:45.158064 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:56.156751 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:46:56.156710 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:47:01.042839 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:01.042805 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff"] Nov 03 17:47:01.054943 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:01.054911 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b19cd46-89hff"] Nov 03 17:47:02.467376 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:02.467338 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb63d12b-f45d-4c6b-9a27-061588712180" path="/var/lib/kubelet/pods/cb63d12b-f45d-4c6b-9a27-061588712180/volumes" Nov 03 17:47:16.029418 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:16.029379 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp"] Nov 03 17:47:16.041384 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:16.041347 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f668a8b-g6ktp"] Nov 03 17:47:16.467748 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:16.467670 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d20866-8b2f-44e8-8579-0cae266cb3d6" path="/var/lib/kubelet/pods/b8d20866-8b2f-44e8-8579-0cae266cb3d6/volumes" Nov 03 17:47:17.324074 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:17.324041 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:47:19.976028 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:19.975994 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:47:26.156679 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:47:26.156640 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:47:33.283520 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:47:33.283476 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:47:56.156147 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:47:56.156098 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:48:19.196386 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:19.196308 4539 scope.go:122] "RemoveContainer" containerID="8dc6a28f6fdee9618f4bda12800cacefc45ce758eeb710218f504cf2c2dc39b5" Nov 03 17:48:19.205319 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:19.205300 4539 scope.go:122] "RemoveContainer" containerID="d724dca2f5b414df3eb06791c7c4fa0b00e8b7d30eb76cd4c5a1aff02acf8b48" Nov 03 17:48:19.391236 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:19.391205 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:24.940636 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:24.940603 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:25.133643 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:25.133608 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:25.281574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:25.281542 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:26.157445 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:48:26.157394 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:48:44.865862 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:48:44.865830 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:56.228087 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:48:56.227971 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:49:17.537297 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:49:17.537269 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:49:17.562043 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:49:17.562010 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:49:19.498980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:49:19.498943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:26.195113 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:49:26.195070 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:49:32.289647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:49:32.289571 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:56.161233 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:49:56.161192 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:50:10.027574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:50:10.027530 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw"] Nov 03 17:50:10.038763 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:50:10.038735 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sglcw"] Nov 03 17:50:10.468743 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:50:10.468667 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f39a3df-801f-4386-924e-81e61157e2df" path="/var/lib/kubelet/pods/1f39a3df-801f-4386-924e-81e61157e2df/volumes" Nov 03 17:50:19.733292 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:50:19.733258 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:22.468962 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:50:22.468917 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:26.158895 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:50:26.158850 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:50:56.217452 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:50:56.217373 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:51:19.257512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:51:19.255594 4539 scope.go:122] "RemoveContainer" containerID="6359f9a50d8106efcdf188fb8917e578a5786a665bad8969bf1124ba845b6fdf" Nov 03 17:51:19.938231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:51:19.938199 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:26.155985 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:51:26.155945 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:51:56.157559 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:51:56.157515 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:52:20.056442 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:20.056394 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:26.155791 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:52:26.155752 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:52:37.090638 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:37.090551 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:37.615488 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:37.615410 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:37.975899 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:37.975806 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:41.723402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:41.723370 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:41.899833 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:41.899791 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:43.144150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:43.144114 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:43.365827 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:43.365797 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:43.534146 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:43.534105 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:43.705831 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:43.705796 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:45.307921 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:45.307883 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:47.092637 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:47.092600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:48.793462 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:48.793385 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:50.502475 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:50.502444 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:52.182468 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:52.182418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:53.998368 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:53.998326 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:55.715107 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:55.715033 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:56.155531 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:52:56.155490 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:52:57.041915 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:57.041879 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:57.213840 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:57.213797 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:57.440989 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:57.440906 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:59.249120 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.249079 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:59.269670 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.269641 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx"] Nov 03 17:52:59.270073 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.270061 4539 cpu_manager.go:414] "containerMap: removing container" podUID="1f39a3df-801f-4386-924e-81e61157e2df" containerName="revoke-keys" Nov 03 17:52:59.270118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.270075 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f39a3df-801f-4386-924e-81e61157e2df" containerName="revoke-keys" Nov 03 17:52:59.270118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.270089 4539 cpu_manager.go:414] "containerMap: removing container" podUID="68bda8e6-f896-4ada-85b8-0178831756c5" containerName="main" Nov 03 17:52:59.270118 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.270094 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bda8e6-f896-4ada-85b8-0178831756c5" containerName="main" Nov 03 17:52:59.270221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.270174 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f39a3df-801f-4386-924e-81e61157e2df" containerName="revoke-keys" Nov 03 17:52:59.270221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.270183 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bda8e6-f896-4ada-85b8-0178831756c5" containerName="main" Nov 03 17:52:59.274017 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.273988 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.303161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.303130 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-27d3f0\"/\"openshift-service-ca.crt\"" Nov 03 17:52:59.303359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.303248 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-27d3f0\"/\"kube-root-ca.crt\"" Nov 03 17:52:59.306067 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.306041 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx"] Nov 03 17:52:59.366627 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.366586 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.366843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.366672 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjrz\" (UniqueName: \"kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.366843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.366780 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.366974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.366875 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.366974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.366915 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.366974 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.366951 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.367121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.367033 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-27d3f0\"/\"simulator-single-main-27d3f0-kserve-self-signed-certs\"" Nov 03 17:52:59.367121 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.367093 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-27d3f0\"/\"default-dockercfg-66sx4\"" Nov 03 17:52:59.414370 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.414330 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:59.468255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468220 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468255 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468264 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468309 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468347 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjrz\" (UniqueName: \"kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468363 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468556 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468511 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468808 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.468847 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468846 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.469035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.468885 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.470815 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.470790 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.471020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.471004 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.514223 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.514128 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjrz\" (UniqueName: \"kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz\") pod \"simulator-single-main-27d3f0-kserve-77bddff458-zdkvx\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.605590 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.605554 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:52:59.921684 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:52:59.921655 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx"] Nov 03 17:52:59.929241 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:52:59.927149 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c73b9b0_3232_455a_b4a0_dfd263466a28.slice/crio-3093af15f57ed85e3f0369f4533e34556bb8f683ac13c068458e7cd313359940 WatchSource:0}: Error finding container 3093af15f57ed85e3f0369f4533e34556bb8f683ac13c068458e7cd313359940: Status 404 returned error can't find the container with id 3093af15f57ed85e3f0369f4533e34556bb8f683ac13c068458e7cd313359940 Nov 03 17:53:00.627502 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:00.627462 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" event={"ID":"3c73b9b0-3232-455a-b4a0-dfd263466a28","Type":"ContainerStarted","Data":"563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f"} Nov 03 17:53:00.627980 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:00.627517 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" event={"ID":"3c73b9b0-3232-455a-b4a0-dfd263466a28","Type":"ContainerStarted","Data":"3093af15f57ed85e3f0369f4533e34556bb8f683ac13c068458e7cd313359940"} Nov 03 17:53:00.671716 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:00.671649 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" podStartSLOduration=1.671627833 podStartE2EDuration="1.671627833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:00.66957021 +0000 UTC m=+5924.362841238" watchObservedRunningTime="2026-11-03 17:53:00.671627833 +0000 UTC m=+5924.364898857" Nov 03 17:53:01.097100 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:01.097061 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:01.606034 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:01.606001 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:53:02.785329 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:02.785282 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:04.504984 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:04.504943 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:06.192628 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:06.192589 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:07.908574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:07.908534 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:08.720739 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:08.720707 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:09.606455 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:09.606393 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:53:09.619533 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:09.619508 4539 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:53:09.691473 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:09.691435 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:53:09.754764 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:09.754733 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:11.395592 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:11.395555 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:13.111454 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:13.111361 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:14.893676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:14.893635 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:16.608286 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:16.608251 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:18.386064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:18.386027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:20.100990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:20.100952 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:20.242576 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:20.242547 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:21.805113 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:21.805076 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:23.583646 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:23.583608 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:25.286543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:25.286509 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:26.156252 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:53:26.156216 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:53:26.982574 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:26.982536 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:28.683364 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:28.683322 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:30.395321 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:30.395289 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:32.186649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:32.186609 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:33.983860 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:33.983825 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:35.688117 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:35.688086 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:37.384745 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:37.384710 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:39.101824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:39.101788 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:40.788919 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:40.788858 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:41.535800 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:41.535772 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:41.987923 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:41.987830 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:42.578477 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:42.578418 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:44.463070 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:44.463027 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:46.189247 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:46.189213 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:46.467990 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:46.467898 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx"] Nov 03 17:53:46.572674 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:46.572638 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:47.997676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:47.997641 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:48.607887 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:48.607855 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:48.750385 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:48.750340 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:49.633911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:49.633864 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:54.391814 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:54.391771 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:54.554664 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:53:54.554629 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:56.155897 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:53:56.155858 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:54:01.489024 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.488915 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" podUID="3c73b9b0-3232-455a-b4a0-dfd263466a28" containerName="main" containerID="cri-o://563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f" gracePeriod=45 Nov 03 17:54:01.748268 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.748196 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:54:01.882678 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.882646 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm\") pod \"3c73b9b0-3232-455a-b4a0-dfd263466a28\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " Nov 03 17:54:01.882875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.882758 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir\") pod \"3c73b9b0-3232-455a-b4a0-dfd263466a28\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " Nov 03 17:54:01.882875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.882802 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache\") pod \"3c73b9b0-3232-455a-b4a0-dfd263466a28\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " Nov 03 17:54:01.882875 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.882838 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs\") pod \"3c73b9b0-3232-455a-b4a0-dfd263466a28\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " Nov 03 17:54:01.883064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.882873 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz\" (UniqueName: \"kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz\") pod \"3c73b9b0-3232-455a-b4a0-dfd263466a28\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " Nov 03 17:54:01.883064 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.882922 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home\") pod \"3c73b9b0-3232-455a-b4a0-dfd263466a28\" (UID: \"3c73b9b0-3232-455a-b4a0-dfd263466a28\") " Nov 03 17:54:01.883231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.883073 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir" pod "3c73b9b0-3232-455a-b4a0-dfd263466a28" (UID: "3c73b9b0-3232-455a-b4a0-dfd263466a28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:01.883231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.883136 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache" pod "3c73b9b0-3232-455a-b4a0-dfd263466a28" (UID: "3c73b9b0-3232-455a-b4a0-dfd263466a28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:01.883231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.883200 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home" pod "3c73b9b0-3232-455a-b4a0-dfd263466a28" (UID: "3c73b9b0-3232-455a-b4a0-dfd263466a28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:01.883359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.883283 4539 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-tmp-dir\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:54:01.883359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.883306 4539 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:54:01.884993 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.884967 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm" pod "3c73b9b0-3232-455a-b4a0-dfd263466a28" (UID: "3c73b9b0-3232-455a-b4a0-dfd263466a28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:01.885085 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.885065 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs" pod "3c73b9b0-3232-455a-b4a0-dfd263466a28" (UID: "3c73b9b0-3232-455a-b4a0-dfd263466a28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:01.885178 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.885160 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz" pod "3c73b9b0-3232-455a-b4a0-dfd263466a28" (UID: "3c73b9b0-3232-455a-b4a0-dfd263466a28"). InnerVolumeSpecName "kube-api-access-tkjrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:01.961527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.961488 4539 generic.go:356] "Generic (PLEG): container finished" podID="3c73b9b0-3232-455a-b4a0-dfd263466a28" containerID="563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f" exitCode=0 Nov 03 17:54:01.961737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.961540 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" Nov 03 17:54:01.961737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.961578 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" event={"ID":"3c73b9b0-3232-455a-b4a0-dfd263466a28","Type":"ContainerDied","Data":"563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f"} Nov 03 17:54:01.961737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.961634 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" event={"ID":"3c73b9b0-3232-455a-b4a0-dfd263466a28","Type":"ContainerDied","Data":"3093af15f57ed85e3f0369f4533e34556bb8f683ac13c068458e7cd313359940"} Nov 03 17:54:01.961737 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.961658 4539 scope.go:122] "RemoveContainer" containerID="563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f" Nov 03 17:54:01.971624 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.971599 4539 scope.go:122] "RemoveContainer" containerID="563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f" Nov 03 17:54:01.971930 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:54:01.971907 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f\": container with ID starting with 563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f not found: ID does not exist" containerID="563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f" Nov 03 17:54:01.972020 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.971938 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f"} err="failed to get container status \"563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f\": rpc error: code = NotFound desc = could not find container \"563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f\": container with ID starting with 563d29313707ff6bcba4705d79abd910e83bc26cb6e071ddfd55a72daa30738f not found: ID does not exist" Nov 03 17:54:01.983841 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.983820 4539 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c73b9b0-3232-455a-b4a0-dfd263466a28-tls-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:54:01.983929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.983842 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjrz\" (UniqueName: \"kubernetes.io/projected/3c73b9b0-3232-455a-b4a0-dfd263466a28-kube-api-access-tkjrz\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:54:01.983929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.983852 4539 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-home\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:54:01.983929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:01.983861 4539 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c73b9b0-3232-455a-b4a0-dfd263466a28-dshm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:54:02.012917 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:02.012851 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx"] Nov 03 17:54:02.019086 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:02.019064 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx"] Nov 03 17:54:02.467402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:02.467368 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c73b9b0-3232-455a-b4a0-dfd263466a28" path="/var/lib/kubelet/pods/3c73b9b0-3232-455a-b4a0-dfd263466a28/volumes" Nov 03 17:54:02.681480 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:02.681409 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-27d3f0/simulator-single-main-27d3f0-kserve-77bddff458-zdkvx" podUID="3c73b9b0-3232-455a-b4a0-dfd263466a28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.216:8000/ready\": context deadline exceeded" Nov 03 17:54:17.574991 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:17.574960 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:54:17.598598 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:17.598571 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:54:20.241982 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:20.241929 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:20.399526 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:20.399491 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:21.558736 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:21.558697 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:21.766578 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:21.766543 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:21.916877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:21.916789 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:22.154754 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:22.154722 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:22.659456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:22.659406 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:23.589265 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:23.589233 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:24.503499 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:24.503383 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:25.417648 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:25.417614 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:26.158223 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:54:26.158185 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:54:27.202044 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:27.201993 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:28.802676 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:28.802635 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:30.594457 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:30.594415 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:30.844221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:30.844182 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:30.988998 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:30.988908 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:32.395248 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:32.395213 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:34.122174 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:34.122143 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:35.797846 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:35.797813 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:37.483584 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:37.483552 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:39.217512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:39.217483 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:40.880979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:40.880947 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:41.028938 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:41.028902 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:42.358877 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:42.358836 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:42.878315 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:42.878281 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:43.030894 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:43.030863 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:43.383604 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:54:43.383569 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:56.158468 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:54:56.158416 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:55:02.275550 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.275514 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6"] Nov 03 17:55:02.276162 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.275747 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" podUID="a5a057da-36a9-417d-a66d-3f56cc6c6fff" containerName="manager" containerID="cri-o://b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b" gracePeriod=10 Nov 03 17:55:02.531869 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.531797 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 17:55:02.674162 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.674133 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p\" (UniqueName: \"kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p\") pod \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " Nov 03 17:55:02.674340 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.674264 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume\") pod \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\" (UID: \"a5a057da-36a9-417d-a66d-3f56cc6c6fff\") " Nov 03 17:55:02.674647 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.674624 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume" pod "a5a057da-36a9-417d-a66d-3f56cc6c6fff" (UID: "a5a057da-36a9-417d-a66d-3f56cc6c6fff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:02.676527 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.676411 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p" pod "a5a057da-36a9-417d-a66d-3f56cc6c6fff" (UID: "a5a057da-36a9-417d-a66d-3f56cc6c6fff"). InnerVolumeSpecName "kube-api-access-99c5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:02.775337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.775300 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99c5p\" (UniqueName: \"kubernetes.io/projected/a5a057da-36a9-417d-a66d-3f56cc6c6fff-kube-api-access-99c5p\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:55:02.775337 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:02.775331 4539 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5a057da-36a9-417d-a66d-3f56cc6c6fff-extensions-socket-volume\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:55:03.324979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.324939 4539 generic.go:356] "Generic (PLEG): container finished" podID="a5a057da-36a9-417d-a66d-3f56cc6c6fff" containerID="b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b" exitCode=0 Nov 03 17:55:03.325407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.324990 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" Nov 03 17:55:03.325407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.325035 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" event={"ID":"a5a057da-36a9-417d-a66d-3f56cc6c6fff","Type":"ContainerDied","Data":"b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b"} Nov 03 17:55:03.325407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.325083 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6" event={"ID":"a5a057da-36a9-417d-a66d-3f56cc6c6fff","Type":"ContainerDied","Data":"a183e32c11f668ad90caec9e1bc66e33bf8aee5dffa03186d5ab1f9760a65a7e"} Nov 03 17:55:03.325407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.325110 4539 scope.go:122] "RemoveContainer" containerID="b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b" Nov 03 17:55:03.337665 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.337648 4539 scope.go:122] "RemoveContainer" containerID="b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b" Nov 03 17:55:03.337924 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:55:03.337902 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b\": container with ID starting with b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b not found: ID does not exist" containerID="b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b" Nov 03 17:55:03.337965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.337935 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b"} err="failed to get container status \"b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b\": rpc error: code = NotFound desc = could not find container \"b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b\": container with ID starting with b86afa20cb445f3f0cea3925599da141c63de2aab0d5736313826120e0a0f92b not found: ID does not exist" Nov 03 17:55:03.388725 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.388698 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6"] Nov 03 17:55:03.400943 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:03.400920 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtqt6"] Nov 03 17:55:04.467630 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:04.467599 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a057da-36a9-417d-a66d-3f56cc6c6fff" path="/var/lib/kubelet/pods/a5a057da-36a9-417d-a66d-3f56cc6c6fff/volumes" Nov 03 17:55:20.378965 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:20.378924 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:24.397131 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:24.397040 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:24.561959 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:24.561920 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:26.159628 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:55:26.159592 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:55:41.320734 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:55:41.320645 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:56.156909 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:55:56.156867 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:56:05.428649 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.428615 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5"] Nov 03 17:56:05.429164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.429017 4539 cpu_manager.go:414] "containerMap: removing container" podUID="3c73b9b0-3232-455a-b4a0-dfd263466a28" containerName="main" Nov 03 17:56:05.429164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.429028 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c73b9b0-3232-455a-b4a0-dfd263466a28" containerName="main" Nov 03 17:56:05.429164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.429049 4539 cpu_manager.go:414] "containerMap: removing container" podUID="a5a057da-36a9-417d-a66d-3f56cc6c6fff" containerName="manager" Nov 03 17:56:05.429164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.429054 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a057da-36a9-417d-a66d-3f56cc6c6fff" containerName="manager" Nov 03 17:56:05.429164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.429115 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c73b9b0-3232-455a-b4a0-dfd263466a28" containerName="main" Nov 03 17:56:05.429164 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.429126 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a057da-36a9-417d-a66d-3f56cc6c6fff" containerName="manager" Nov 03 17:56:05.432279 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.432252 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.449504 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.449476 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5"] Nov 03 17:56:05.459461 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.459417 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vfdxp\"" Nov 03 17:56:05.467373 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.467342 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4fbw\" (UniqueName: \"kubernetes.io/projected/9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528-kube-api-access-w4fbw\") pod \"kuadrant-operator-controller-manager-678d87766f-n65w5\" (UID: \"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.467522 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.467501 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-n65w5\" (UID: \"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.568035 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.567995 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-n65w5\" (UID: \"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.568231 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.568088 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fbw\" (UniqueName: \"kubernetes.io/projected/9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528-kube-api-access-w4fbw\") pod \"kuadrant-operator-controller-manager-678d87766f-n65w5\" (UID: \"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.568415 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.568394 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-n65w5\" (UID: \"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.592911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.592880 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fbw\" (UniqueName: \"kubernetes.io/projected/9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528-kube-api-access-w4fbw\") pod \"kuadrant-operator-controller-manager-678d87766f-n65w5\" (UID: \"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:05.745119 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:05.745029 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:06.073290 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:06.073255 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5"] Nov 03 17:56:06.075799 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:56:06.075769 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5a7a2b_de4e_4ec0_8409_0b5b71ce1528.slice/crio-d44fd1f861c056ed16dd96205096655cd55f94eb5c7c38c5befc5f1c3e06690f WatchSource:0}: Error finding container d44fd1f861c056ed16dd96205096655cd55f94eb5c7c38c5befc5f1c3e06690f: Status 404 returned error can't find the container with id d44fd1f861c056ed16dd96205096655cd55f94eb5c7c38c5befc5f1c3e06690f Nov 03 17:56:06.655557 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:06.655512 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" event={"ID":"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528","Type":"ContainerStarted","Data":"37cf57b86534a6483fd8dca391a0794e4c70958a0b2017a3c3b765c70f93aa6a"} Nov 03 17:56:06.656072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:06.655581 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" event={"ID":"9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528","Type":"ContainerStarted","Data":"d44fd1f861c056ed16dd96205096655cd55f94eb5c7c38c5befc5f1c3e06690f"} Nov 03 17:56:06.656072 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:06.655609 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:06.714519 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:06.714451 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" podStartSLOduration=1.714404034 podStartE2EDuration="1.714404034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:06.712340147 +0000 UTC m=+6110.405611168" watchObservedRunningTime="2026-11-03 17:56:06.714404034 +0000 UTC m=+6110.407675120" Nov 03 17:56:11.326195 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:11.326152 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:11.463245 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:11.463215 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:17.664256 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:17.664225 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-n65w5" Nov 03 17:56:20.458643 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:20.458596 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:26.162678 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:56:26.162632 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:56:45.619387 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:45.619244 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:56:45.620391 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:45.620261 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:56.063539 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.063499 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6"] Nov 03 17:56:56.067161 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.067136 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.075402 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.075374 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2zv\" (UniqueName: \"kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.075569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.075441 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.075569 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.075489 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.075666 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.075640 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.098359 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.098326 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7vlf\"" Nov 03 17:56:56.098635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.098395 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:56:56.098635 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.098477 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 17:56:56.099999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.099976 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6"] Nov 03 17:56:56.158807 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:56:56.158772 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:56:56.176351 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.176319 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.176512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.176381 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.176512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.176458 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.176512 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.176488 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2zv\" (UniqueName: \"kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.177205 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.177181 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.179197 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.179176 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.179268 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.179208 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.200619 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.200584 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2zv\" (UniqueName: \"kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv\") pod \"maas-api-74d6c75f5f-nm2v6\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.381150 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.381062 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:56.681168 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.681121 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6"] Nov 03 17:56:56.683945 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:56:56.683916 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdfc1da_3887_47a8_892c_0fc1f752ba88.slice/crio-6f1281c65615d5e99eb6173ac54158be2a7a4f2a824a01288166fa4490d76853 WatchSource:0}: Error finding container 6f1281c65615d5e99eb6173ac54158be2a7a4f2a824a01288166fa4490d76853: Status 404 returned error can't find the container with id 6f1281c65615d5e99eb6173ac54158be2a7a4f2a824a01288166fa4490d76853 Nov 03 17:56:56.685188 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.685172 4539 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:56:56.962338 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:56.962245 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" event={"ID":"9bdfc1da-3887-47a8-892c-0fc1f752ba88","Type":"ContainerStarted","Data":"6f1281c65615d5e99eb6173ac54158be2a7a4f2a824a01288166fa4490d76853"} Nov 03 17:56:57.969406 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:57.969369 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" event={"ID":"9bdfc1da-3887-47a8-892c-0fc1f752ba88","Type":"ContainerStarted","Data":"3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a"} Nov 03 17:56:57.969824 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:57.969464 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:56:58.014215 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:56:58.014165 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" podStartSLOduration=1.617176044 podStartE2EDuration="2.014152511s" totalImagesPullingTime="396.976467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:58.013866982 +0000 UTC m=+6161.707138004" watchObservedRunningTime="2026-11-03 17:56:58.014152511 +0000 UTC m=+6161.707423534" Nov 03 17:57:02.416745 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:02.416655 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:03.542336 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.542301 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6"] Nov 03 17:57:03.542843 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.542535 4539 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" podUID="9bdfc1da-3887-47a8-892c-0fc1f752ba88" containerName="maas-api" containerID="cri-o://3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a" gracePeriod=30 Nov 03 17:57:03.547929 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.547898 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:57:03.594306 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.594276 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk"] Nov 03 17:57:03.597861 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.597844 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.630296 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.630268 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk"] Nov 03 17:57:03.650062 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.650012 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9a678c-e264-4e28-915d-18ba3a45d32a-maas-api-tls\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.650264 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.650124 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a678c-e264-4e28-915d-18ba3a45d32a-openshift-service-ca\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.650264 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.650215 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9a678c-e264-4e28-915d-18ba3a45d32a-metrics-certs\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.650264 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.650253 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2fgk\" (UniqueName: \"kubernetes.io/projected/7e9a678c-e264-4e28-915d-18ba3a45d32a-kube-api-access-s2fgk\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.751514 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.751477 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9a678c-e264-4e28-915d-18ba3a45d32a-maas-api-tls\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.751721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.751546 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a678c-e264-4e28-915d-18ba3a45d32a-openshift-service-ca\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.751721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.751610 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9a678c-e264-4e28-915d-18ba3a45d32a-metrics-certs\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.751721 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.751633 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2fgk\" (UniqueName: \"kubernetes.io/projected/7e9a678c-e264-4e28-915d-18ba3a45d32a-kube-api-access-s2fgk\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.752243 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.752214 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a678c-e264-4e28-915d-18ba3a45d32a-openshift-service-ca\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.754194 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.754165 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9a678c-e264-4e28-915d-18ba3a45d32a-metrics-certs\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.754540 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.754519 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9a678c-e264-4e28-915d-18ba3a45d32a-maas-api-tls\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.775971 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.775940 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2fgk\" (UniqueName: \"kubernetes.io/projected/7e9a678c-e264-4e28-915d-18ba3a45d32a-kube-api-access-s2fgk\") pod \"maas-api-78d9c46c8c-c4dkk\" (UID: \"7e9a678c-e264-4e28-915d-18ba3a45d32a\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.800486 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.800409 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:57:03.852377 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.852340 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs\") pod \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " Nov 03 17:57:03.852697 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.852666 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv\" (UniqueName: \"kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv\") pod \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " Nov 03 17:57:03.852834 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.852810 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca\") pod \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " Nov 03 17:57:03.852905 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.852850 4539 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls\") pod \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\" (UID: \"9bdfc1da-3887-47a8-892c-0fc1f752ba88\") " Nov 03 17:57:03.853239 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.853204 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca" pod "9bdfc1da-3887-47a8-892c-0fc1f752ba88" (UID: "9bdfc1da-3887-47a8-892c-0fc1f752ba88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:03.854902 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.854872 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs" pod "9bdfc1da-3887-47a8-892c-0fc1f752ba88" (UID: "9bdfc1da-3887-47a8-892c-0fc1f752ba88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:03.855063 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.855028 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv" pod "9bdfc1da-3887-47a8-892c-0fc1f752ba88" (UID: "9bdfc1da-3887-47a8-892c-0fc1f752ba88"). InnerVolumeSpecName "kube-api-access-zf2zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:03.855294 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.855277 4539 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls" pod "9bdfc1da-3887-47a8-892c-0fc1f752ba88" (UID: "9bdfc1da-3887-47a8-892c-0fc1f752ba88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:03.930911 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.930877 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:03.955617 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.955578 4539 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-metrics-certs\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:57:03.955806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.955621 4539 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2zv\" (UniqueName: \"kubernetes.io/projected/9bdfc1da-3887-47a8-892c-0fc1f752ba88-kube-api-access-zf2zv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:57:03.955806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.955638 4539 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bdfc1da-3887-47a8-892c-0fc1f752ba88-openshift-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:57:03.955806 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:03.955653 4539 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bdfc1da-3887-47a8-892c-0fc1f752ba88-maas-api-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\"" Nov 03 17:57:04.002999 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.002962 4539 generic.go:356] "Generic (PLEG): container finished" podID="9bdfc1da-3887-47a8-892c-0fc1f752ba88" containerID="3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a" exitCode=0 Nov 03 17:57:04.003179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.003005 4539 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" Nov 03 17:57:04.003179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.003047 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" event={"ID":"9bdfc1da-3887-47a8-892c-0fc1f752ba88","Type":"ContainerDied","Data":"3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a"} Nov 03 17:57:04.003179 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.003160 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6" event={"ID":"9bdfc1da-3887-47a8-892c-0fc1f752ba88","Type":"ContainerDied","Data":"6f1281c65615d5e99eb6173ac54158be2a7a4f2a824a01288166fa4490d76853"} Nov 03 17:57:04.003353 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.003192 4539 scope.go:122] "RemoveContainer" containerID="3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a" Nov 03 17:57:04.014095 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.014074 4539 scope.go:122] "RemoveContainer" containerID="3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a" Nov 03 17:57:04.014396 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:57:04.014376 4539 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a\": container with ID starting with 3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a not found: ID does not exist" containerID="3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a" Nov 03 17:57:04.014507 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.014407 4539 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a"} err="failed to get container status \"3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a\": rpc error: code = NotFound desc = could not find container \"3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a\": container with ID starting with 3736a8179a41d32e92266fcace92ac36d6b925dc39f1c59d814f3316852e309a not found: ID does not exist" Nov 03 17:57:04.057979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.057907 4539 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6"] Nov 03 17:57:04.062686 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.062661 4539 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-nm2v6"] Nov 03 17:57:04.229221 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.229193 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk"] Nov 03 17:57:04.231757 ip-10-0-139-190 kubenswrapper[4539]: W1103 17:57:04.231732 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9a678c_e264_4e28_915d_18ba3a45d32a.slice/crio-13f4717ae3b5524618ca41bd11f52b04252802670f379eec10d34641312a95ea WatchSource:0}: Error finding container 13f4717ae3b5524618ca41bd11f52b04252802670f379eec10d34641312a95ea: Status 404 returned error can't find the container with id 13f4717ae3b5524618ca41bd11f52b04252802670f379eec10d34641312a95ea Nov 03 17:57:04.467543 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:04.467464 4539 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdfc1da-3887-47a8-892c-0fc1f752ba88" path="/var/lib/kubelet/pods/9bdfc1da-3887-47a8-892c-0fc1f752ba88/volumes" Nov 03 17:57:05.010888 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:05.010853 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" event={"ID":"7e9a678c-e264-4e28-915d-18ba3a45d32a","Type":"ContainerStarted","Data":"5fe5bfbe851e66b4087052110495732fe03f298f53fc5799417e1b3eeaf62230"} Nov 03 17:57:05.011407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:05.010906 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" event={"ID":"7e9a678c-e264-4e28-915d-18ba3a45d32a","Type":"ContainerStarted","Data":"13f4717ae3b5524618ca41bd11f52b04252802670f379eec10d34641312a95ea"} Nov 03 17:57:05.011407 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:05.010959 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:05.054710 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:05.054659 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" podStartSLOduration=1.755408453 podStartE2EDuration="2.054644785s" totalImagesPullingTime="299.236332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:05.053865589 +0000 UTC m=+6168.747136611" watchObservedRunningTime="2026-11-03 17:57:05.054644785 +0000 UTC m=+6168.747915807" Nov 03 17:57:11.023674 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:11.023636 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-c4dkk" Nov 03 17:57:20.588979 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:57:20.588945 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:26.156234 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:57:26.156197 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:57:56.156156 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:57:56.156112 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:58:20.715640 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:58:20.715600 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:26.156552 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:58:26.156517 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:58:56.155864 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:58:56.155825 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:59:17.609444 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:59:17.609386 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:59:17.634456 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:59:17.634409 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gl2kp_5ad67a8a-e10b-4af1-99ee-d97585bd6b6b/ovn-acl-logging/0.log" Nov 03 17:59:20.769525 ip-10-0-139-190 kubenswrapper[4539]: I1103 17:59:20.769485 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:26.156552 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:59:26.156515 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 17:59:56.198572 ip-10-0-139-190 kubenswrapper[4539]: E1103 17:59:56.198530 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:00:15.636523 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:00:15.635559 4539 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-drxvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:00:15.636523 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:00:15.635672 4539 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-drxvj" podUID="352815c1-13a2-49fc-8075-743b28e27b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:00:20.966482 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:00:20.966412 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:26.171057 ip-10-0-139-190 kubenswrapper[4539]: E1103 18:00:26.170960 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:00:56.225951 ip-10-0-139-190 kubenswrapper[4539]: E1103 18:00:56.225877 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:01:08.129649 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:01:08.129594 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:08.284087 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:01:08.284046 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:21.061884 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:01:21.061843 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:26.163807 ip-10-0-139-190 kubenswrapper[4539]: E1103 18:01:26.163758 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:01:56.156403 ip-10-0-139-190 kubenswrapper[4539]: E1103 18:01:56.156364 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:02:08.904936 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:08.904894 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6b4p_ff6e4b8d-1c33-4e59-9aad-512e10134470/manager/0.log" Nov 03 18:02:09.106963 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:09.106925 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kswm7_22bcb2db-3d3f-49cb-b196-de4ca60476ef/manager/0.log" Nov 03 18:02:09.315003 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:09.314975 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2bgbv_a88f549e-c107-4cb0-a806-5b9dcc15c271/manager/0.log" Nov 03 18:02:09.796743 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:09.796712 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5lqff_8f60d60e-9283-4377-b974-fe3e97be1bff/server/0.log" Nov 03 18:02:09.979620 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:09.979583 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-625qt_e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd/manager/0.log" Nov 03 18:02:11.091068 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:11.091038 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78d9c46c8c-c4dkk_7e9a678c-e264-4e28-915d-18ba3a45d32a/maas-api/0.log" Nov 03 18:02:15.006158 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:15.006065 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-87c98_ae3fc69f-1702-4875-89d2-6ac92f1d5bd8/postgres/0.log" Nov 03 18:02:17.090718 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:17.090683 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mqxzs_dfbb04a9-f7d4-4438-9465-867b1f6ab446/manager/0.log" Nov 03 18:02:17.287225 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:17.287192 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jz5xc_9f80cd1d-c291-40d4-866b-6b1ee9005dac/manager/0.log" Nov 03 18:02:17.672435 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:17.672395 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-n65w5_9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528/manager/0.log" Nov 03 18:02:18.107507 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:18.107477 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5ldg2_6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b/manager/0.log" Nov 03 18:02:19.040037 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:19.040001 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-swdzr_c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0/istio-proxy/0.log" Nov 03 18:02:19.978225 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:19.978188 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7dnkg_929514d9-b731-4a30-8a67-6d1522c756cc/istio-proxy/0.log" Nov 03 18:02:20.603861 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:20.603831 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-drxvj_352815c1-13a2-49fc-8075-743b28e27b10/istio-proxy/0.log" Nov 03 18:02:21.175664 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:21.175625 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:21.613294 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:21.613261 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zvhjn_74c46d2b-77a9-4239-a40c-2b2152f04e00/istio-proxy/0.log" Nov 03 18:02:21.830432 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:21.830390 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z_055bb54a-1e09-47bf-930a-1e90666cb38c/istio-proxy/0.log" Nov 03 18:02:26.155194 ip-10-0-139-190 kubenswrapper[4539]: E1103 18:02:26.155153 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:02:34.827846 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:34.827808 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:34.987122 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:34.987083 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:36.250354 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:36.250318 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:39.419396 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:39.419367 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h6g2k_e0165af3-4e86-4907-8919-6ec892f905b7/global-pull-secret-syncer/0.log" Nov 03 18:02:40.112517 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:40.112463 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" Nov 03 18:02:40.602475 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:40.602444 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-190.ec2.internal_9f77174fff7f21e40f2a31f54c08055a/haproxy/0.log" Nov 03 18:02:42.562157 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:42.562125 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6b4p_ff6e4b8d-1c33-4e59-9aad-512e10134470/manager/0.log" Nov 03 18:02:42.678044 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:42.678010 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kswm7_22bcb2db-3d3f-49cb-b196-de4ca60476ef/manager/0.log" Nov 03 18:02:43.312494 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:43.312456 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2bgbv_a88f549e-c107-4cb0-a806-5b9dcc15c271/manager/0.log" Nov 03 18:02:43.465780 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:43.465754 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5lqff_8f60d60e-9283-4377-b974-fe3e97be1bff/server/0.log" Nov 03 18:02:43.561017 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:43.560983 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-625qt_e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd/manager/0.log" Nov 03 18:02:45.154048 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:45.154011 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6b4p_ff6e4b8d-1c33-4e59-9aad-512e10134470/manager/0.log" Nov 03 18:02:45.279236 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:45.279194 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kswm7_22bcb2db-3d3f-49cb-b196-de4ca60476ef/manager/0.log" Nov 03 18:02:45.925806 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:45.925772 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2bgbv_a88f549e-c107-4cb0-a806-5b9dcc15c271/manager/0.log" Nov 03 18:02:46.098688 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:46.098654 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5lqff_8f60d60e-9283-4377-b974-fe3e97be1bff/server/0.log" Nov 03 18:02:46.222075 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:46.221993 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-625qt_e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd/manager/0.log" Nov 03 18:02:47.783806 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:47.783773 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b6b4p_ff6e4b8d-1c33-4e59-9aad-512e10134470/manager/0.log" Nov 03 18:02:47.862918 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:47.862890 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mqxzs_dfbb04a9-f7d4-4438-9465-867b1f6ab446/manager/0.log" Nov 03 18:02:47.918990 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:47.918957 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kswm7_22bcb2db-3d3f-49cb-b196-de4ca60476ef/manager/0.log" Nov 03 18:02:47.984860 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:47.984834 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jz5xc_9f80cd1d-c291-40d4-866b-6b1ee9005dac/manager/0.log" Nov 03 18:02:48.439490 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:48.439457 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-n65w5_9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528/manager/0.log" Nov 03 18:02:48.659553 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:48.659521 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2bgbv_a88f549e-c107-4cb0-a806-5b9dcc15c271/manager/0.log" Nov 03 18:02:48.762237 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:48.762147 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5ldg2_6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b/manager/0.log" Nov 03 18:02:48.940167 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:48.940140 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5lqff_8f60d60e-9283-4377-b974-fe3e97be1bff/server/0.log" Nov 03 18:02:49.035545 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:49.035513 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-625qt_e4f1bf7d-8cc6-40ff-8d61-4f4940f80afd/manager/0.log" Nov 03 18:02:50.445693 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.445659 4539 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h"] Nov 03 18:02:50.446097 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.446052 4539 cpu_manager.go:414] "containerMap: removing container" podUID="9bdfc1da-3887-47a8-892c-0fc1f752ba88" containerName="maas-api" Nov 03 18:02:50.446097 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.446064 4539 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdfc1da-3887-47a8-892c-0fc1f752ba88" containerName="maas-api" Nov 03 18:02:50.446179 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.446133 4539 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdfc1da-3887-47a8-892c-0fc1f752ba88" containerName="maas-api" Nov 03 18:02:50.449320 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.449304 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.466493 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.466464 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mm7hs\"/\"openshift-service-ca.crt\"" Nov 03 18:02:50.478734 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.478622 4539 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mm7hs\"/\"kube-root-ca.crt\"" Nov 03 18:02:50.478734 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.478679 4539 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mm7hs\"/\"default-dockercfg-z6f5n\"" Nov 03 18:02:50.481331 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.481308 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h"] Nov 03 18:02:50.581110 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.581084 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-podres\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.581314 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.581119 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-lib-modules\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.581314 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.581151 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-sys\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.581314 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.581274 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-proc\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.581507 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.581328 4539 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdg8\" (UniqueName: \"kubernetes.io/projected/2693462c-0379-4203-95ff-83b20adaaf0f-kube-api-access-8zdg8\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.682803 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682763 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-sys\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.682982 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682814 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-proc\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.682982 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682886 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdg8\" (UniqueName: \"kubernetes.io/projected/2693462c-0379-4203-95ff-83b20adaaf0f-kube-api-access-8zdg8\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.682982 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682886 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-sys\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.682982 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682944 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-podres\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.682982 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682952 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-proc\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.683291 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.682994 4539 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-lib-modules\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.683291 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.683001 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-podres\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.683291 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.683066 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2693462c-0379-4203-95ff-83b20adaaf0f-lib-modules\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.710833 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.710767 4539 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdg8\" (UniqueName: \"kubernetes.io/projected/2693462c-0379-4203-95ff-83b20adaaf0f-kube-api-access-8zdg8\") pod \"perf-node-gather-daemonset-fqq7h\" (UID: \"2693462c-0379-4203-95ff-83b20adaaf0f\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:50.737668 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.737644 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-swdzr_c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0/istio-proxy/0.log" Nov 03 18:02:50.761923 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:50.761898 4539 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:51.059799 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:51.059771 4539 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h"] Nov 03 18:02:51.068312 ip-10-0-139-190 kubenswrapper[4539]: W1103 18:02:51.068275 4539 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2693462c_0379_4203_95ff_83b20adaaf0f.slice/crio-e8e7bc1cc20aadaf6295431a61746ca3683ef896ed87aae1ebf77883330b2f6a WatchSource:0}: Error finding container e8e7bc1cc20aadaf6295431a61746ca3683ef896ed87aae1ebf77883330b2f6a: Status 404 returned error can't find the container with id e8e7bc1cc20aadaf6295431a61746ca3683ef896ed87aae1ebf77883330b2f6a Nov 03 18:02:51.206805 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:51.206763 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" event={"ID":"2693462c-0379-4203-95ff-83b20adaaf0f","Type":"ContainerStarted","Data":"e8e7bc1cc20aadaf6295431a61746ca3683ef896ed87aae1ebf77883330b2f6a"} Nov 03 18:02:52.214499 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.214464 4539 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" event={"ID":"2693462c-0379-4203-95ff-83b20adaaf0f","Type":"ContainerStarted","Data":"8f54843c36897f648be90cf0aa1edd56937c22b6ac8c4a90e36d926a15288c85"} Nov 03 18:02:52.214955 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.214556 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:52.258098 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.258029 4539 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" podStartSLOduration=2.258011459 podStartE2EDuration="2.258011459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:52.256360291 +0000 UTC m=+6515.949631313" watchObservedRunningTime="2026-11-03 18:02:52.258011459 +0000 UTC m=+6515.951282482" Nov 03 18:02:52.559386 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.559343 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrj9b_4b0a4814-d478-4c0a-8d41-9fb5a015cc4f/node-exporter/0.log" Nov 03 18:02:52.669509 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.669474 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrj9b_4b0a4814-d478-4c0a-8d41-9fb5a015cc4f/kube-rbac-proxy/0.log" Nov 03 18:02:52.672881 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.672861 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7dnkg_929514d9-b731-4a30-8a67-6d1522c756cc/istio-proxy/0.log" Nov 03 18:02:52.778733 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.778702 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrj9b_4b0a4814-d478-4c0a-8d41-9fb5a015cc4f/init-textfile/0.log" Nov 03 18:02:52.849074 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:52.848999 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-drxvj_352815c1-13a2-49fc-8075-743b28e27b10/istio-proxy/0.log" Nov 03 18:02:53.147437 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:53.147347 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zvhjn_74c46d2b-77a9-4239-a40c-2b2152f04e00/istio-proxy/0.log" Nov 03 18:02:53.205321 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:53.205291 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z_055bb54a-1e09-47bf-930a-1e90666cb38c/istio-proxy/0.log" Nov 03 18:02:53.950562 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:53.950536 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rx9n2_72f619ef-f2cb-4d6b-a106-d8abdf2c766c/prometheus-operator/0.log" Nov 03 18:02:54.065329 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:54.065301 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rx9n2_72f619ef-f2cb-4d6b-a106-d8abdf2c766c/kube-rbac-proxy/0.log" Nov 03 18:02:54.186933 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:54.186899 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-twdlk_dbe0478a-bf9d-47a2-b337-4bea66f9d737/prometheus-operator-admission-webhook/0.log" Nov 03 18:02:54.301006 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:54.300962 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5576bdf989-bqn8b_f93fc47d-fdb2-4f53-b45a-3023e5b82889/telemeter-client/0.log" Nov 03 18:02:54.414151 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:54.414124 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5576bdf989-bqn8b_f93fc47d-fdb2-4f53-b45a-3023e5b82889/reload/0.log" Nov 03 18:02:54.527985 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:54.527960 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5576bdf989-bqn8b_f93fc47d-fdb2-4f53-b45a-3023e5b82889/kube-rbac-proxy/0.log" Nov 03 18:02:56.155782 ip-10-0-139-190 kubenswrapper[4539]: E1103 18:02:56.155748 4539 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4vn9v" containerName="extractor" Nov 03 18:02:57.417726 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:57.417694 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mqxzs_dfbb04a9-f7d4-4438-9465-867b1f6ab446/manager/0.log" Nov 03 18:02:57.477770 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:57.477745 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jz5xc_9f80cd1d-c291-40d4-866b-6b1ee9005dac/manager/0.log" Nov 03 18:02:57.622192 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:57.622163 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-n65w5_9e5a7a2b-de4e-4ec0-8409-0b5b71ce1528/manager/0.log" Nov 03 18:02:57.819565 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:57.819530 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5ldg2_6cfff5e2-954c-4ec8-87d0-5d3c61e33e3b/manager/0.log" Nov 03 18:02:58.232327 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:58.232249 4539 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-fqq7h" Nov 03 18:02:58.653988 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:02:58.653958 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-vmkds_56a09c07-a809-456c-912a-2697b95e1c63/download-server/0.log" Nov 03 18:03:00.269006 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:00.268978 4539 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvp5p_8bf7557b-7e3c-49a9-a522-1e46ba83b79f/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:02.167881 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:02.167847 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lv59d_49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960/dns/0.log" Nov 03 18:03:02.273855 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:02.273822 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lv59d_49aa209e-24d5-4e0a-9e4a-a2cf9d2f6960/kube-rbac-proxy/0.log" Nov 03 18:03:02.606296 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:02.606259 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7dwpd_cac12a2a-91d1-4812-bf79-8d4ab8bb4a8c/dns-node-resolver/0.log" Nov 03 18:03:03.872588 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:03.872556 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-swdzr_c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0/istio-proxy/0.log" Nov 03 18:03:04.043995 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:04.043962 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66d98664dc-l7jt8_21f70bc9-9ef1-4a86-9dbb-b513dca3ab28/registry/0.log" Nov 03 18:03:04.334610 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:04.334543 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wmmj_d5a6779b-623f-4618-bc63-f9efc8008fa4/node-ca/0.log" Nov 03 18:03:05.810166 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:05.810129 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7dnkg_929514d9-b731-4a30-8a67-6d1522c756cc/istio-proxy/0.log" Nov 03 18:03:06.051512 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:06.051476 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-drxvj_352815c1-13a2-49fc-8075-743b28e27b10/istio-proxy/0.log" Nov 03 18:03:06.115487 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:06.115350 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-swdzr_c1d72e5f-33ca-45e2-b4dd-bd70d15ab4e0/istio-proxy/0.log" Nov 03 18:03:06.435549 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:06.435364 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zvhjn_74c46d2b-77a9-4239-a40c-2b2152f04e00/istio-proxy/0.log" Nov 03 18:03:06.497636 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:06.497604 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-49d4z_055bb54a-1e09-47bf-930a-1e90666cb38c/istio-proxy/0.log" Nov 03 18:03:08.749101 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:08.749070 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7dnkg_929514d9-b731-4a30-8a67-6d1522c756cc/istio-proxy/0.log" Nov 03 18:03:09.096701 ip-10-0-139-190 kubenswrapper[4539]: I1103 18:03:09.096662 4539 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-drxvj_352815c1-13a2-49fc-8075-743b28e27b10/istio-proxy/0.log"